starlingx/docs
Code Issues Proposed changes

Cert-Manager Use for StarlingX Platform Services

Browse Source 
Initial draft procedures.
Resolve merge conflicts.
Incorporate patchset 1 review comments.
Incorporate patchset 2 review comments.
Incorporate patchset 3 review comments.
Incorporate patchset 4 review comments. Open questions for J. Sun to be addressed.
Incorporate patchset 5 review comments.
Made sample url used in overrides generic.
Incorporate patchset 8 review comments.
Added note about issuer_root_ca recommended by J. Sun.
Incorporate patchset 10 review comments.
Fix formatting issue in output.
Incorporate patchset 12 review comments.

Story: 2007361
Task: 42625

Signed-off-by: Ron Stone <ronald.stone@windriver.com>
Change-Id: I5a73f902902acc02baccb92995f696a4b19fb773
This commit is contained in:
Ron Stone 2021-11-09 10:48:34 -05:00
parent baff361d45
commit 3e03a0bc82
11 changed files with 745 additions and 176 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

0
doc/source/_includes/rest-api-and-docker-registry-certificate-d9c4e8587419.rest Normal file
View File

97
doc/source/security/kubernetes/configure-docker-registry-certificate-after-installation-c519edbfe90a.rst Normal file
View File

@ -0,0 +1,97 @@
.. _configure-docker-registry-certificate-after-installation-c519edbfe90a:
=====================================
Configure Docker Registry Certificate
=====================================
.. rubric:: |context|
The local Docker registry provides secure HTTPS access using the registry API.
.. rubric:: |context|
By default, a self-signed server certificate is generated at installation time
for the registry API. For more secure access, an intermediate or Root CA-signed
server certificate is strongly recommended.
To configure or update the HTTPS certificate for the local Docker registry,
create a certificate named ``system-registry-local-certificate`` in the
``deployment`` namespace. The ``secretName`` attribute of this certificate's
spec must also be named ``system-registry-local-certificate``.
See the example procedure below for creating the certificate for the local
Docker registry. This example assumes you have configured a
``system-local-ca`` ClusterIssuer as described in
:ref:`starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834`.
Update the following fields:
* The ``duration`` and ``renewBefore`` dates for the expiry and renewal times
you desire. The system will automatically renew and re-install the
certificate.
* The ``subject`` fields to identify your particular system.
* The ``ipAddresses`` with the |OAM| Floating IP Address and the MGMT Floating
IP address for this system. Use the :command:`system addrpool-list` command
to get the |OAM| floating IP Address and management floating IP Address for
your system.
* The ``dnsNames`` with ``registry.local``, ``registry.central`` and any |FQDN|
names configured for this system's |OAM| Floating IP Address in an external
DNS server.
.. rubric:: |proc|
#. Create the Docker certificate yaml configuration file.
.. code-block::
~(keystone_admin)]$ cat <<EOF > docker-certificate.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: system-registry-local-certificate
namespace: deployment
spec:
secretName: system-registry-local-certificate
issuerRef:
name: system-local-ca
kind: ClusterIssuer
duration: 2160h # 90d
renewBefore: 360h # 15d
subject:
organizations:
- ABC-Company
organizationalUnits:
- StarlingX-system-registry-local
ipAddresses:
- <OAM_FLOATING_IP>
- <MGMT_FLOATING_IP>
dnsNames:
- registry.local
- registry.central
- <external-FQDN-for-OAM-Floating-IP-Address, if applicable>
#. Apply the configuration.
.. code-block::
~(keystone_admin)]$ kubectl apply -f docker-certificate.yaml
#. Verify the configuration.
.. code-block::
~(keystone_admin)]$ kubectl get certificate system-registry-local-certificate –n deployment
If configuration was successful, the certificate’s Ready status will be
``True``.
.. rubric:: |result|
The Docker registry certificate installation is now complete, and Cert-Manager
will handle the lifecycle management of the certificate.

367
doc/source/security/kubernetes/configure-oidc-auth-applications.rst
View File

@ -2,24 +2,27 @@
.. cwn1581381515361
.. _configure-oidc-auth-applications:
================================
Configure OIDC Auth Applications
================================
=============================
Set up OIDC Auth Applications
=============================
The **oidc-auth-apps** application is a system application that needs to be
configured to use a remote Windows Active Directory server to authenticate
users of the Kubernetes API. The **oidc-auth-apps** is packaged in the ISO
users of the Kubernetes API. The ``oidc-auth-apps`` is packaged in the ISO
and uploaded by default.
Configure OIDC Auth Applications
================================
.. rubric:: |prereq|
.. _configure-oidc-auth-applications-ul-gpz-x51-llb:
- You must have configured the Kubernetes **kube-apiserver** to use
- You must have configured the Kubernetes ``kube-apiserver`` to use
the **oidc-auth-apps** |OIDC| identity provider for validation of
tokens in Kubernetes API requests, which use |OIDC| authentication. For
more information on configuring the Kubernetes **kube-apiserver**, see
more information on configuring the Kubernetes ``kube-apiserver``, see
:ref:`Configure Kubernetes for OIDC Token Validation while
Bootstrapping the System
<configure-kubernetes-for-oidc-token-validation-while-bootstrapping-the-system>`
@ -27,81 +30,245 @@ and uploaded by default.
Bootstrapping the System
<configure-kubernetes-for-oidc-token-validation-after-bootstrapping-the-system>`.
- You must have a |CA| signed certificate \(dex-cert.pem file\), and private
key \(dex-key.pem file\) for the dex |OIDC| Identity Provider of
**oidc-auth-apps**.
This certificate *must* have the |prod|'s floating |OAM| IP Address in
the |SAN| list. If you are planning on defining and using a DNS
name for the |prod|'s floating |OAM| IP Address, then this DNS name
*must* also be in the |SAN| list. Refer to the documentation for
the external |CA| that you are using, in order to create a signed
certificate and key.
If you are using an intermediate |CA| to sign the dex certificate, include
both the dex certificate \(signed by the intermediate |CA|\), and the
intermediate |CA|'s certificate \(signed by the Root |CA|\) in that order,
in **dex-cert.pem**.
- You must have the certificate of the |CA|\(**dex-ca.pem** file\) that
signed the above certificate for the dex |OIDC| Identity Provider of
**oidc-auth-apps**.
If an intermediate |CA| was used to sign the dex certificate and both the
dex certificate and the intermediate |CA| certificate was included in
**dex-cert.pem**, then the **dex-ca.pem** file should contain the root
|CA|'s certificate.
If the signing |CA| \(**dex-ca.pem**\) is not a well-known trusted |CA|,
you must ensure the system trusts the |CA| by specifying it either during
the bootstrap phase of system installation, by specifying '**ssl\_ca\_cert:
dex-ca.pem**' in the ansible bootstrap overrides **localhost.yml** file, or
by using the **system certificate-install -m ssl\_ca dex-ca.pem** command.
.. rubric:: |proc|
#. Create certificates using one of the following options.
.. _configure-oidc-auth-applications-steps-kll-nbm-tkb:
#. Create certificates using cert-manager (recommended):
#. Create the secret, **local-dex.tls**, with the certificate and key, to be
used by the **oidc-auth-apps**, as well as the secret, **dex-client-secret**,
with the |CA|'s certificate that signed the **local-dex.tls** certificate.
Certificates used by ``oidc-auth-apps`` can be managed by Cert-Manager.
Doing so will automatically renew the certificates before they expire.
The ``system-local-ca`` ClusterIssuer (see
:ref:`starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834`)
will be used to issue this certificate.
For example, assuming the cert and key pem files for creating these
secrets are in /home/sysadmin/ssl/, run the following commands to create
the secrets:
.. important::
The namespace for ``oidc-auth-apps`` must be ``kube-system``.
.. note::
**oidc-auth-apps** looks specifically for secrets of these names in
the **kube-system** namespace.
#. Create the |OIDC| client and identity provider server certificate and
private key pair.
For the generic secret **dex-client-secret**, the filename must be
'**dex-ca.pem**'.
.. code-block:: none
~(keystone_admin)]$ cat <<EOF > oidc-auth-apps-certificate.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: oidc-auth-apps-certificate
namespace: kube-system
spec:
secretName: oidc-auth-apps-certificate
duration: 2160h # 90 days
renewBefore: 360h # 15 days
issuerRef:
name: system-local-ca
kind: ClusterIssuer
commonName: <OAM_floating_IP_address>
organizations:
- ABC-Company
organizationalUnits:
- StarlingX-system-oidc-auth-apps
ipAddresses:
- <OAM_floating_IP_address>
EOF
#. Apply the configuration.
.. code-block:: none
~(keystone_admin)]$ kubectl apply -f oidc-auth-apps-certificate.yaml
#. Verify the configuration.
.. code-block:: none
~(keystone_admin)]$ kubectl get certificate oidc-auth-apps-certificate –n kube-system
#. Configure the |OIDC|-client with both the |OIDC| Client and Identity
Server Certificate and the |OIDC| Client and Identity Trusted |CA|
certificate.
Create a secret with the certificate of the root |CA| that signed the
|OIDC| client and identity provider's server certificate. In this
example, it will be the ``ca.crt`` of the ``system-local-ca``
ClusterIssuer).
.. code-block:: none
~(keystone_admin)]$ mkdir /home/sysadmin/ssl
~(keystone_admin)]$ kubectl get secret system-local-ca -n cert-manager -o=jsonpath='{.data.ca\.crt}' | base64 --decode > /home/sysadmin/ssl/dex-ca-cert.crt
~(keystone_admin)]$ kubectl create secret generic dex-ca-cert --from-file=/home/sysadmin/ssl/dex-ca-cert.pem -n kube-system
~(keystone_admin)]$ cat <<EOF > stx-oidc-client.yaml
tlsName: oidc-auth-apps-certificate
config:
# The OIDC-client container mounts the dex-ca-cert secret at /home, therefore
# issuer_root_ca: /home/<filename-only-of-generic-secret>
issuer_root_ca: /home/dex-ca-cert.crt
issuer_root_ca_secret: dex-ca-cert
EOF
~(keystone_admin)]$ system helm-override-update oidc-auth-apps oidc-client kube-system --values stx-oidc-client.yaml
#. Create a secret with the certificate of the |CA| that signed the
certificate of the remote Windows Active Directory server that you
will be using.
Create the secret ``wad-ca-cert`` with the |CA|'s certificate that
signed the Active Directory's certificate using the following
command:
.. code-block:: none
~(keystone_admin)]$ kubectl create secret generic wad-ca-cert.crt --from-file=ssl/wad-ca-cert -n kube-system
Add the following sections to your dex helm overrides to configure the
|OIDC| Client and Identity Server Certificate and the Windows Active
Directory server |CA| Certificate for the |OIDC| Identity Server:
.. code-block:: none
certs:
web:
secret:
tlsName: oidc-auth-apps-certificate
caName: oidc-auth-apps-certificate
grpc:
secret:
serverTlsName: oidc-auth-apps-certificate
clientTlsName: oidc-auth-apps-certificate
caName: oidc-auth-apps-certificate
extraVolumes:
- name: certdir
secret:
secretName: wad-ca-cert
extraVolumeMounts:
- name: certdir
mountPath: /etc/ssl/certs/wad-ca-cert.crt
#. Apply the overrides configuration.
.. code-block:: none
~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values dex-overrides.yaml
#. Configure the secret observer to track changes.
Change the cronSchedule according to your needs. The cronSchedule
controls how often the application checks to see if the certificate
mounted on the dex and oidc-client pods had changed.
Create a YAML configuration to modify the cronSchedule according to
your needs.
The cronSchedule controls how often the application checks to see
if the certificate mounted on the dex and oidc-client pods changed.
The following example sets the schedule to every 15 minutes.
.. code-block:: none
~(keystone_admin)]$ cat <<EOF > secret-observer-overrides.yaml
cronSchedule: "*/15 * * * *"
observedSecrets:
- secretName: "dex-ca-cert"
filename: "dex-ca-cert.crt"
deploymentToRestart: "stx-oidc-client"
- secretName: "oidc-auth-apps-certificate"
filename: "tls.crt"
deploymentToRestart: "stx-oidc-client"
- secretName: "oidc-auth-apps-certificate"
filename: "tls.crt"
deploymentToRestart: "oidc-dex"
EOF
Execute the following command to update the overrides:
.. code-block:: none
~(keystone_admin)]$ system helm-override-update oidc-auth-apps secret-observer kube-system --values secret-observer-overrides.yaml
#. Use certificates generated and signed by an external |CA|.
.. rubric:: |prereq|
- You must have a |CA| signed certificate (``dex-cert.pem`` file), and
private key (``dex-key.pem file``) for the dex |OIDC| Identity
Provider of **oidc-auth-apps**.
This certificate *must* have the |prod|'s floating |OAM| IP Address
in the |SAN| list. If you are planning on defining and using a DNS
name for the |prod|'s floating |OAM| IP Address, then this DNS name
*must* also be in the |SAN| list. Refer to the documentation for the
external |CA| that you are using, in order to create a signed
certificate and key.
If you are using an intermediate |CA| to sign the dex certificate,
include both the dex certificate (signed by the intermediate |CA|),
and the intermediate |CA|'s certificate (signed by the Root |CA|) in
that order, in ``dex-cert.pem``.
- You must have the certificate of the |CA| (``dex-ca.pem`` file) that
signed the above certificate for the dex |OIDC| Identity Provider of
**oidc-auth-apps**.
If an intermediate |CA| was used to sign the dex certificate and both
the dex certificate and the intermediate |CA| certificate was
included in ``dex-cert.pem``, then the ``dex-ca.pem`` file should
contain the root |CA|'s certificate.
If the signing |CA| (``dex-ca.pem``) is not a well-known trusted
|CA|, you must ensure the system trusts the |CA| by specifying it
either during the bootstrap phase of system installation, by
specifying ``ssl_ca_cert: dex-ca.pem`` in the ansible bootstrap
overrides ``localhost.yml`` file, or by using the :command:`system
certificate-install -m ssl_ca dex-ca.pem` command.
#. Create the secret, ``local-dex.tls``, with the certificate and key,
to be used by the **oidc-auth-apps**, as well as the secret,
``dex-client-secret``, with the |CA|'s certificate that signed the
``local-dex.tls`` certificate.
For example, assuming the cert and key pem files for creating these
secrets are in ``/home/sysadmin/ssl/``, run the following commands to
create the secrets:
.. note::
**oidc-auth-apps** looks specifically for secrets of these names
in the ``kube-system`` namespace.
For the generic secret ``dex-client-secret``, the filename must
be ``dex-ca.pem``.
.. code-block:: none
~(keystone_admin)]$ kubectl create secret tls local-dex.tls --cert=ssl/dex-cert.pem --key=ssl/dex-key.pem -n kube-system
~(keystone_admin)]$ kubectl create secret generic dex-client-secret --from-file=/home/sysadmin/ssl/dex-ca.pem -n kube-system
Create the secret ``wad-ca-cert`` with the |CA|'s certificate that signed
the Active Directory's certificate using the following command:
.. code-block:: none
~(keystone_admin)]$ kubectl create secret generic wad-ca-cert --from-file=ssl/wad-ca-cert.crt -n kube-system
#. Specify user overrides for **oidc-auth-apps** application, by using the
following command:
.. code-block:: none
~(keystone_admin)]$ kubectl create secret tls local-dex.tls --cert=ssl/dex-cert.pem --key=ssl/dex-key.pem -n kube-system
~(keystone_admin)]$ kubectl create secret generic dex-client-secret --from-file=/home/sysadmin/ssl/dex-ca.pem -n kube-system
Create the secret **wadcert** with the |CA|'s certificate that signed
the Active Directory's certificate using the following command:
.. code-block:: none
~(keystone_admin)]$ kubectl create secret generic wadcert --from-file=ssl/AD_CA.cer -n kube-system
#. Specify user overrides for **oidc-auth-apps** application, by using the following command:
.. code-block:: none
~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/dex-overrides.yaml
~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/dex-overrides.yaml --reuse-values
The dex-overrides.yaml file contains the desired dex helm chart overrides
\(that is, the |LDAP| connector configuration for the Active Directory
service, optional token expiry, and so on.\), and volume mounts for
providing access to the **wadcert** secret, described in this section.
(that is, the |LDAP| connector configuration for the Active Directory
service, optional token expiry, and so on), and volume mounts for
providing access to the ``wadcert`` secret, described in this section.
For the complete list of dex helm chart values supported, see `Dex Helm
Chart Values
@ -112,15 +279,15 @@ and uploaded by default.
The example below configures a token expiry of ten hours, a single |LDAP|
connector to an Active Directory service using HTTPS \(LDAPS\) using the
**wadcert** secret configured in this section, the required Active
``wadcert`` secret configured in this section, the required Active
Directory service login information \(that is, bindDN, and bindPW\), and
example :command:`userSearch`, and :command:`groupSearch` clauses.
\(Optional\) There is a default secret in the dex configuration for
**staticClients**. You can change this using helm overrides. For example,
(Optional) There is a default secret in the dex configuration for
``staticClients``. You can change this using helm overrides. For example,
to change the secret, first run the following command to see the default
settings. In this example, 10.10.10.2 is the |prod-long| |OAM| floating IP
address.
settings. In this example, ``10.10.10.2`` is the |prod-long| |OAM| floating
IP address.
.. code-block:: none
@ -136,15 +303,17 @@ and uploaded by default.
Change the secret from the output and copy the entire configuration section
shown above in to your dex overrides file shown in the example below.
.. note::
Do NOT forget to include the id, name, and redirectURIs parameters.
.. warning::
Do not forget to include the id, name, and redirectURIs parameters.
.. note::
There is an internal **client\_secret** that is used between the
There is an internal ``client_secret`` that is used between the
oidc-client container and the dex container. It is recommended that you
configure a unique, more secure **client\_secret** by specifying the
configure a unique, more secure ``client_secret`` by specifying the
value in the dex overrides file, as shown in the example below.
.. begin-connector-config
.. code-block:: none
config:
@ -161,22 +330,22 @@ and uploaded by default.
name: OpenLDAP
id: ldap
config:
host: pv-windows-acti.cumulus.wrs.com:636
rootCA: /etc/ssl/certs/adcert/AD_CA.cer
host: pv-windows-acti.windows-activedir.example.com:636
rootCA: /etc/ssl/certs/adcert/wad-ca-cert.crt
insecureNoSSL: false
insecureSkipVerify: false
bindDN: cn=Administrator,cn=Users,dc=cumulus,dc=wrs,dc=com
bindDN: cn=Administrator,cn=Users,dc=windows-activedir,dc=example,dc=com
bindPW: [<password>]
usernamePrompt: Username
userSearch:
baseDN: ou=Users,ou=Titanium,dc=cumulus,dc=wrs,dc=com
baseDN: ou=Users,ou=Titanium,dc=windows-activedir,dc=example,dc=com
filter: "(objectClass=user)"
username: sAMAccountName
idAttr: sAMAccountName
emailAttr: sAMAccountName
nameAttr: displayName
groupSearch:
baseDN: ou=Groups,ou=Titanium,dc=cumulus,dc=wrs,dc=com
baseDN: ou=Groups,ou=Titanium,dc=windows-activedir,dc=example,dc=com
filter: "(objectClass=group)"
userAttr: DN
groupAttr: member
@ -184,29 +353,31 @@ and uploaded by default.
extraVolumes:
- name: certdir
secret:
secretName: wadcert
secretName: wad-ca-cert
extraVolumeMounts:
- name: certdir
mountPath: /etc/ssl/certs/adcert
mountPath: /etc/ssl/certs/wad-ca-cert.crt
.. end-connector-config
If more than one Windows Active Directory service is required for
authenticating the different users of the |prod|, multiple '**ldap**'
authenticating the different users of the |prod|, multiple ``ldap``
type connectors can be configured; one for each Windows Active
Directory service.
If more than one **userSearch** plus **groupSearch** clauses are
If more than one ``userSearch`` plus ``groupSearch`` clauses are
required for the same Windows Active Directory service, multiple
'**ldap**' type connectors, with the same host information but
different **userSearch** plus **groupSearch** clauses, should be used.
``ldap`` type connectors, with the same host information but
different ``userSearch`` plus ``groupSearch`` clauses, should be used.
Whenever you use multiple '**ldap**' type connectors, ensure you use
unique '**name:**' and '**id:**' parameters for each connector.
Whenever you use multiple ``ldap`` type connectors, ensure you use
unique ``name:`` and ``id:`` parameters for each connector.
#. An override in the secrets in the dex helm chart must be accompanied by an
override in the oidc-client helm chart.
#. An override in the secrets in the dex helm chart must be accompanied by
an override in the oidc-client helm chart.
The following override is sufficient for changing the secret in the
/home/sysadmin/oidc-client-overrides.yaml file.
``/home/sysadmin/oidc-client-overrides.yaml`` file.
.. code-block:: none
@ -217,13 +388,13 @@ and uploaded by default.
.. code-block:: none
~(keystone_admin)]$ system helm-override-update oidc-auth-apps oidc-client kube-system --values /home/sysadmin/oidc-client-overrides.yaml
~(keystone_admin)]$ system helm-override-update oidc-auth-apps oidc-client kube-system --values /home/sysadmin/oidc-client-overrides.yaml --reuse-values
.. note::
If you need to manually override the secrets, the client\_secret in the
If you need to manually override the secrets, the client_secret in the
oidc-client overrides must match the staticClients secret and
client\_secret in the dex overrides, otherwise the oidc-auth |CLI|
client_secret in the dex overrides, otherwise the oidc-auth |CLI|
client will not function.
#. Use the :command:`system application-apply` command to apply the
@ -231,4 +402,4 @@ and uploaded by default.
.. code-block:: none
~(keystone_admin)]$ system application-apply oidc-auth-apps
~(keystone_admin)]$ system application-apply oidc-auth-apps

106
doc/source/security/kubernetes/configure-rest-api-applications-and-web-administration-server-certificates-after-installation-6816457ab95f.rst Normal file
View File

@ -0,0 +1,106 @@
.. _configure-rest-api-applications-and-web-administration-server-certificates-after-installation-6816457ab95f:
=========================================================================
Configure REST API Applications and Web Administration Server certificate
=========================================================================
.. rubric:: |context|
|prod| provides support for secure HTTPS external connections used for
StarlingX REST API application endpoints (Keystone, Barbican and StarlingX) and
the |prod| web administration server. By default, HTTPS access to StarlingX
REST and Web Server endpoints is disabled. They are accessible via HTTP only.
To enable secure HTTPS access, an x509 certificate and key must be configured.
You can update the certificate used for HTTPS access at any time.
To configure or update the HTTPS certificate for the StarlingX REST API and Web
Server endpoints, create a certificate named ``system-restapi-gui-certificate``
in the ``deployment`` namespace. The ``secretName`` attribute of this
certificate's spec must also be named ``system-restapi-gui-certificate``.
See the example procedure below for creating the certificate for the StarlingX
REST API and Web Server endpoints. This example assumes you have configured a
``system-local-ca`` ClusterIssuer as described in
:ref:`starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834`.
Update the following fields:
* The ``duration`` and ``renewBefore`` dates for the expiry and renewal times
you desire. The system will automatically renew and re-install the
certificate.
* The ``subject`` fields to identify your particular system.
* The ``ipAddresses`` with the |OAM| Floating IP Address for this system.
* The ``dnsNames`` with any |FQDN| names configured for this system in an
external DNS server.
.. note::
If you plan to use the container-based remote CLIs, due to a limitation in
the Python2 SSL certificate validation, the certificate used for the
'system-restapi-gui-certificate' certificate must either have:
1. CN=IPADDRESS and SANs=IPADDRESS
or
1. CN=FQDN and SANs=FQDN
where IPADDRESS and FQDN are for the |OAM| Floating IP Address.
.. rubric:: |proc|
#. Create the REST API certificate yaml configuration file.
.. code-block::
~(keystone_admin)]$ cat <<EOF > restapi-certificate.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: system-restapi-gui-certificate
namespace: deployment
spec:
secretName: system-restapi-gui-certificate
issuerRef:
name: system-local-ca
kind: ClusterIssuer
duration: 2160h # 90 days
renewBefore: 360h # 15 days
commonName: < oam floating IP Address or FQDN >
subject:
organizations:
- ABC-Company
organizationalUnits:
- StarlingX-system-restapi-gui
ipAddresses:
- < oam floating IP address >
dnsNames:
- < oam floating FQDN >
EOF
#. Apply the configuration.
.. code-block::
~(keystone_admin)]$ kubectl apply -f restapi-certificate.yaml
#. Verify the configuration.
.. code-block::
~(keystone_admin)]$ kubectl get certificate system-restapi-gui-certificate –n deployment
If configuration was successful, the certificate’s Ready status will be
``True``.
.. rubric:: |result|
The REST and Web Server certificate installation is now complete, and
Cert-Manager will handle the lifecycle management of the certificate.

137
doc/source/security/kubernetes/https-access-overview.rst
View File

@ -16,73 +16,76 @@ in the following sections.
.. table::
:widths: auto
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Certificate | Auto Created | Renewal Status |
+===========================================================+=============================================================================+====================================================================================================+
| Kubernetes Root CA Certificate | Yes | NOT AUTO-RENEWED; Default expiry is set at 10 years |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Cluster Admin client certificate used by kubectl | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kube-controller-manager client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kube-scheduler client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kube-apiserver server certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kube-apiserver's kubelet client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kubelet client certificate | Yes | auto-renewed by kubelet feature enabled by default |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| etcd Root CA certificate | Yes | NOT AUTO-RENEWED; Default expiry is set at 10 years |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| etcd server certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| etcd client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| kube-apiserver's etcd client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| StarlingX REST API & HORIZON Server Certificate | Yes (But the auto-created certificate is self-signed and should be changed) | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Local Registry Server Certificate | Yes (But the auto-created certificate is self-signed and should be changed) | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| OIDC Client and Dex Server Server Certificate | No | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| OIDC Client and Dex Server CA certificate | No | NOT AUTO-RENEWED. CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| OIDC Remote WAD CA Certificate | No | NOT AUTO-RENEWED. CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Vault Server Certificate | Yes | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Vault Root CA certificate | Yes | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Portieris Server Certificate | Yes | Auto renewed by cert-manager; BUT COSTOMER MUST restart Portieris after the certificate is renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Portieris remote registry and notary server CA Certificate| No | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Root CA DC Admin Endpoint CA Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Intermediate CA DC Admin Endpoint CA Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| DC Admin Endpoint Server Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| System trusted CA Certificates | No | NOT AUTO-RENEWED as these are certificates that are not necessarily owned by Cloud Platform |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------+
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Certificate | Auto Created | Renewal Status |
+===========================================================+=============================================================================+========================================================================================================+
| Kubernetes Root CA Certificate | Yes | NOT AUTO-RENEWED; Default expiry is set at 10 years; MUST be renewed via CLI. |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Cluster Admin client certificate used by kubectl | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kube-controller-manager client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kube-scheduler client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kube-apiserver server certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kube-apiserver's kubelet client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kubelet client certificate | Yes | auto-renewed by kubelet feature enabled by default |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| etcd Root CA certificate | Yes | NOT AUTO-RENEWED; Default expiry is set at 10 years |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| etcd server certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| etcd client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| kube-apiserver's etcd client certificate | Yes | auto-renewed by cron job |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| StarlingX REST API & HORIZON Server Certificate | Yes (But the auto-created certificate is self-signed and should be changed) | auto-renewed if configured with cert-manager; |
| | | NOT AUTO-RENEWED if configured with :command:`system certificate-install ..`, must be renewed via CLI |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Local Registry Server Certificate | Yes (But the auto-created certificate is self-signed and should be changed) | auto-renewed if configured with cert-manager; |
| | | NOT AUTO-RENEWED if configured with :command:`system certificate-install ..`, must be renewed via CLI |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| OIDC Client and Dex Server Server Certificate | No | auto-renewed if configured with cert-manager; |
| | | NOT AUTO-RENEWED if configured with an externally generated certificate, CUSTOMER MUST renew via CLI. |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| OIDC Client and Dex Server CA certificate | No | NOT AUTO-RENEWED. CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| OIDC Remote WAD CA Certificate | No | NOT AUTO-RENEWED. CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Vault Server Certificate | Yes | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Vault Root CA certificate | Yes | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Portieris Server Certificate | Yes | Auto renewed by cert-manager; BUT CUSTOMER MUST restart Portieris after the certificate is renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Portieris remote registry and notary server CA Certificate| No | NOT AUTO-RENEWED; CUSTOMER MUST renew via CLIs |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Root CA DC Admin Endpoint CA Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| Intermediate CA DC Admin Endpoint CA Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| DC Admin Endpoint Server Certificate | Yes | auto-renewed |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| System trusted CA Certificates | No | NOT AUTO-RENEWED as these are certificates that are not necessarily owned by the platform |
+-----------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
Where:

18
doc/source/security/kubernetes/index.rst
View File

@ -102,10 +102,12 @@ HTTPS Certificate Management
https-access-overview
utility-script-to-display-certificates
starlingx-rest-api-applications-and-the-web-administration-server
starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834
kubernetes-certificates-f4196d7cae9c
etcd-certificates-c1fc943e4a9c
security-install-update-the-docker-registry-certificate
kubernetes-root-ca-certificate
configure-rest-api-applications-and-web-administration-server-certificates-after-installation-6816457ab95f
configure-docker-registry-certificate-after-installation-c519edbfe90a
oidc-client-dex-server-certificates-dc174462d51a
portieris-server-certificate-a0c7054844bd
vault-server-certificate-8573125eeea6
@ -260,6 +262,16 @@ Security Features
security-hardening-firewall-options
isolate-starlingx-internal-cloud-management-network
************************
Deprecated Functionality
************************
.. toctree::
:maxdepth: 1
starlingx-rest-api-applications-and-the-web-administration-server-deprecated
security-install-update-the-docker-registry-certificate-deprecated
***************************************
Appendix: Locally creating certificates
***************************************
@ -268,4 +280,4 @@ Appendix: Locally creating certificates
:maxdepth: 1
create-certificates-locally-using-openssl
create-certificates-locally-using-cert-manager-on-the-controller
create-certificates-locally-using-cert-manager-on-the-controller

24
doc/source/security/kubernetes/oidc-client-dex-server-certificates-dc174462d51a.rst
View File

@ -14,7 +14,7 @@ of tokens.
.. note::
For details on how installing, configuring, and using oidc-auth-apps,
For details on installing, configuring, and using oidc-auth-apps,
refer to :ref:`User Authentication Using Windows Active Directory
<user-authentication-using-windows-active-directory-security-index>`.
@ -41,7 +41,7 @@ This certificate is stored in Kubernetes TLS secret ``local-dex.tls``.
The |OIDC| trusted |CA| certificate is the |CA| certificate that signs the
|OIDC| client and identity server certificate.
It has to be installed for |OIDC| client to verify identity server’s
It has to be installed for |OIDC| client to verify identity server's
certificate for HTTPS connection.
|OIDC| trusted |CA| certificate is stored in Kubernetes secret
@ -54,7 +54,7 @@ Directory that |OIDC| is configured to proxy authentication requests to.
In order for |OIDC| identity provider (as the authentication proxy) to securely
connect and authenticate users to the Windows Active Directory by HTTPS, the
|WAD|’s |CA| certificate needs to installed and configured for |OIDC| to trust
|WAD|'s |CA| certificate needs to installed and configured for |OIDC| to trust
the Windows Active Directory.
-------------------------
@ -74,16 +74,25 @@ Kubernetes secrets.
Update/Renew OIDC certificates
------------------------------
.. warning::
|OIDC| certificates are not auto renewed. They have to be updated manually
by updating the secrets from the new certificate files and restart the
``oidc-auth`` application.
The |OIDC| client and identity provider certificate, if configured via
cert-manager (as described in :ref:`Configure OIDC Auth Applications
<configure-oidc-auth-applications>`), is auto-renewed.
However, the |OIDC| client and identity provider trusted |CA| certificate and
the Windows Active Directory |CA| certificate are not auto renewed. They have
to be renewed manually by updating the secrets from the new certificate files
and restarting the ``oidc-auth`` application.
.. rubric:: |proc|
#. Update/renew |OIDC| client and identity provider server certificate:
.. note::
This step is only required if you are not using cert-manager for your
certificate as described in :ref:`Configure OIDC Auth Applications
<configure-oidc-auth-applications>`.
.. code-block:: none
~(keystone_admin)]$ kubectl create secret tls local-dex.tls --cert=/home/sysadmin/new_ssl/dex-cert.pem --key=/home/sysadmin/new_ssl/dex-key.pem --save-config --dry-run=client -n kube-system -o yaml | kubectl apply -f -
@ -106,4 +115,3 @@ Update/Renew OIDC certificates
~(keystone_admin)]$ kubectl rollout restart deployment oidc-dex -n kube-system
~(keystone_admin)]$ kubectl rollout restart deployment stx-oidc-client -n kube-system

4
doc/source/security/kubernetes/security-install-update-the-docker-registry-certificate.rst → doc/source/security/kubernetes/security-install-update-the-docker-registry-certificate-deprecated.rst
View File

@ -6,6 +6,10 @@
Local Registry Server Certificates
==================================
.. note::
This procedure is deprecated. For up-to-date information, refer to:
:ref:`configure-docker-registry-certificate-after-installation-c519edbfe90a`.
For the Local Docker Registry, HTTPS is always enabled. By default, a
self-signed server certificate and key is generated and installed for this
endpoint. However, it is strongly recommended that you update the server

162
doc/source/security/kubernetes/starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834.rst Normal file
View File

@ -0,0 +1,162 @@
.. _starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834:
========================
Create a local CA Issuer
========================
|org| recommends that a single local |CA| Issuer be created on the platform to
create, sign, and anchor all of your platform certificates. This |CA| can be
either a stand-alone local Root |CA| or a local Intermediate |CA| (whose
certificate is signed by an external Root |CA|). This simplifies your overall
platform certificate configuration, and means that external clients interfacing
with the platform's HTTPS endpoints, need only be given a single Root |CA|
certificate to add to their trusted |CAs|.
Create the ClusterIssuer
========================
Create a local Root CA
----------------------
The following sample procedure illustrates how to create a unique standalone
local Root |CA| (``system-local-ca``) that can be used to create, sign, and
anchor all of your platform certificates.
Update the ``subject`` fields to identify your particular system.
It is recommended that a 3-5 year duration be used for operational simplicity
since, although the certificate will automatically renew locally, when it does
renew, you will need to re-distribute the |CA|'s new public certificate to all
external clients using the platform's HTTPS endpoints.
The created ``system-local-ca`` Root |CA| is cluster-wide, so it can be used to
create all platform certificates and can also be used for hosted applications'
certificates.
.. rubric:: |proc|
#. Create a cluster issuer yaml configuration file.
.. code-block:: none
~(keystone_admin)]$ cat <<EOF > cluster-issuer.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: system-selfsigning
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: system-local-ca
namespace: cert-manager
spec:
subject:
organizations:
- ABC-Company
organizationalUnits:
- StarlingX-system-local-ca
secretName: system-local-ca
commonName: system-local-ca
isCA: true
duration: 43800h # 5 year
renewBefore: 720h # 30 days
issuerRef:
name: system-selfsigning
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: system-local-ca
spec:
ca:
secretName: system-local-ca
EOF
#. Apply the configuration.
.. code-block:: none
~(keystone_admin)]$ kubectl apply –f cluster-issuer.yaml
#. Verify the configuration.
.. code-block::
~(keystone_admin)]$ kubectl get clusterissuer
#. Write the public certificate of this |CA| to a ``.pem`` file that can be
distributed to external clients using the platform's HTTPS endpoints.
.. code-block::
~(keystone_admin)]$ kubectl get secret <secretname> -n <namespacename> -o=jsonpath='{.data.ca\.crt}' | base64 --decode > <pemfilename>
Create a local Intermediate CA
------------------------------
Alternatively, if you are using an external RootCA, the following procedure is
an example of how to create a local Intermediate |CA| (whose certificate is
signed by an external Intermediate or Root |CA|) that can be used to
create, sign, and anchor all of your platform certificates. Refer to the
documentation for your external Intermediate or Root |CA| for information on
how to create a public certificate and private key for an intermediate |CA|.
It is recommended that a 3-5 year duration be used for operational simplicity
since this certificate will need to be manually renewed with the externally
generated certificate and key, and then referenced via the ClusterIssuer's
``spec.ca.secretName``. The |TLS| secret must be created in the Cluster
Resource Namespace, which defaults to ``cert-manager`` on the platform.
The ``system-local-ca`` Root |CA| is cluster-wide, so it can be used to create
all platform certificates and can also be used for hosted applications'
certificates.
#. Copy the |PEM| encoded certificate and key from the externally generated
|CA| to the controller host.
#. Create a |TLS| secret in ‘cert-manager’ namespace with the certificate/Key
files:
.. code-block:: none
~(keystone_admin)]$ kubectl -n cert-manager create secret tls system-local-ca --cert=./cert.pem --key=./key.pem
#. Create ClusterIssuer and the |CA| certificate.
.. code-block:: none
~(keystone_admin)]$ cat <<EOF > cluster-issuer.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: system-local-ca
spec:
ca:
secretName: system-local-ca
EOF
#. Apply the configuration.
.. code-block:: none
~(keystone_admin)]$ kubectl apply –f cluster-issuer.yaml
#. Verify the configuration.
.. code-block::
~(keystone_admin)]$ kubectl get clusterissuer
If the configuration is successful, the clusterissuer for
``system-local-ca`` will have Ready status of ``True``.
The clusterissuer is now ready to issue certificates on the platform.

4
doc/source/security/kubernetes/starlingx-rest-api-applications-and-the-web-administration-server.rst → doc/source/security/kubernetes/starlingx-rest-api-applications-and-the-web-administration-server-deprecated.rst
View File

@ -6,6 +6,10 @@
StarlingX REST API Applications and the Web Administration Server Certificate
=============================================================================
.. note::
This procedure is deprecated. For up-to-date information, refer to:
:ref:`starlingx-rest-api-applications-and-the-web-admin-server-cert-9196c5794834`.
By default, |prod| provides HTTP access to REST API application endpoints
\(Keystone, Barbican and |prod|\) and the web administration server. For
improved security, you can enable HTTPS access. When HTTPS access is

2
doc/source/shared/abbrevs.txt
View File

@ -29,6 +29,8 @@
.. |CNI| replace:: :abbr:`CNI (Container Networking Interface)`
.. |CNIs| replace:: :abbr:`CNIs (Container Networking Interfaces)`
.. |CoW| replace:: :abbr:`CoW (Copy on Write)`
.. |CRD| replace:: :abbr:`CRD (Custom Resource Definition)`
.. |CRDs| replace:: :abbr:`CRDs (Custom Resource Definitions)`
.. |CSK| replace:: :abbr:`CSK (Code Signing Key)`
.. |CSKs| replace:: :abbr:`CSKs (Code Signing Keys)`
.. |CVE| replace:: :abbr:`CVE (Common Vulnerabilities and Exposures)`