|
|
@ -12,64 +12,89 @@ |
|
|
|
License for the specific language governing permissions and limitations |
|
|
|
under the License. |
|
|
|
|
|
|
|
Config Qinling with existing Kubernetes cluster |
|
|
|
=============================================== |
|
|
|
Config Qinling with existing kubernetes/etcd cluster |
|
|
|
==================================================== |
|
|
|
|
|
|
|
In most cases, it's not ideal to set up a new dedicated Kubernetes cluster for |
|
|
|
Qinling. The component which works with Kubernetes cluster in Qinling is the |
|
|
|
``qinling-engine``. Follow the steps below to configure Qinling to work with an |
|
|
|
existing Kubernetes cluster, and make Qinling access the Kubernetes API with |
|
|
|
authentication and authorization. |
|
|
|
In most cases, it's not ideal to set up a new dedicated kubernetes cluster for |
|
|
|
Qinling. The component which works with kubernetes cluster in Qinling is the |
|
|
|
``qinling-engine``. Follow the steps below to configure Qinling to work with |
|
|
|
existing kubernetes/etcd cluster, and make Qinling access the kubernetes/etcd |
|
|
|
service with authentication and authorization. |
|
|
|
|
|
|
|
Qinling Configurations |
|
|
|
Prerequisites |
|
|
|
~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
* You know the kubernetes API address and etcd service address, for example: |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
K8S_ADDRESS=10.0.0.5 |
|
|
|
ETCD_ADDRESS=10.0.0.6 |
|
|
|
|
|
|
|
.. end |
|
|
|
|
|
|
|
* You have CA certificates of the kubernetes and etcd respectively and store on |
|
|
|
the host that ``qinling-engine`` is running. |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
K8S_CA_CERT=$HOME/ca.crt |
|
|
|
K8S_CA_KEY=$HOME/ca.key |
|
|
|
ETCD_CA_CERT=$HOME/etcd_ca.crt |
|
|
|
ETCD_CA_KEY=$HOME/etcd_ca.key |
|
|
|
|
|
|
|
.. end |
|
|
|
|
|
|
|
* This guide assumes |
|
|
|
`RBAC <https://kubernetes.io/docs/admin/authorization/rbac/>`_ is enabled in |
|
|
|
the kubernetes cluster. |
|
|
|
|
|
|
|
Qinling configurations |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
Below are the options that relate to accessing the Kubernetes API in Qinling's |
|
|
|
configuration file, all of them are under the ``kubernetes`` section. |
|
|
|
Below are the options and their default values that relate to accessing the |
|
|
|
Kubernetes API and etcd in Qinling's configuration file. |
|
|
|
|
|
|
|
.. code-block:: ini |
|
|
|
|
|
|
|
[kubernetes] |
|
|
|
kube_host = http://127.0.0.1:8001 |
|
|
|
kube_host = https://127.0.0.1:8001 |
|
|
|
use_api_certificate = True |
|
|
|
ssl_ca_cert = /etc/qinling/pki/kubernetes/ca.crt |
|
|
|
cert_file = /etc/qinling/pki/kubernetes/qinling.crt |
|
|
|
key_file = /etc/qinling/pki/kubernetes/qinling.key |
|
|
|
|
|
|
|
For now, just update the ``kube_host`` to the URI which the Kubernetes API |
|
|
|
serves for HTTPS connections with authentication and authorization, for |
|
|
|
example, ``kube_host = https://kube-api.example.com:6443``. We will cover the |
|
|
|
other options in the following sections. |
|
|
|
[etcd] |
|
|
|
host = 127.0.0.1 |
|
|
|
port = 2379 |
|
|
|
protocol = https |
|
|
|
ca_cert = /etc/qinling/pki/etcd/ca.crt |
|
|
|
cert_file = /etc/qinling/pki/etcd/qinling-etcd-client.crt |
|
|
|
cert_key = /etc/qinling/pki/etcd/qinling-etcd-client.key |
|
|
|
|
|
|
|
Authentication and Authorization |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
.. end |
|
|
|
|
|
|
|
The access to the Kubernetes API is controlled by several modules, refer to |
|
|
|
`Controlling Access to the Kubernetes API <https://kubernetes.io/docs/admin/accessing-the-api/>`_ |
|
|
|
for more details. |
|
|
|
Change the kubernetes and etcd service addresses: |
|
|
|
|
|
|
|
By default, Qinling engine is configured to access the Kubernetes API with |
|
|
|
a client certificate for authentication(``use_api_certificate`` is set to |
|
|
|
``True``), so make sure that the Kubernetes API server is running with the |
|
|
|
``--client-ca-file=SOMEFILE`` option for client certificate authentication to |
|
|
|
be enabled. The common name of the subject in the client certificate is used as |
|
|
|
the user name for the requests that Qinling engine makes to the Kubernetes API |
|
|
|
server. Refer to |
|
|
|
`Authentication in Kubernetes <https://kubernetes.io/docs/admin/authentication/>`_. |
|
|
|
.. code-block:: ini |
|
|
|
|
|
|
|
If `RBAC Authorization <https://kubernetes.io/docs/admin/authorization/rbac/>`_ |
|
|
|
is enabled in the Kubernetes API, we will also have to grant access to |
|
|
|
resources in Kubernetes for the specific user that Qinling uses to make |
|
|
|
requests to the Kubernetes API. Using RBAC Authorization can ensure that |
|
|
|
Qinling access the Kubernetes API with only the permission that it needs. |
|
|
|
[kubernetes] |
|
|
|
kube_host = https://${K8S_ADDRESS}:6443 |
|
|
|
... |
|
|
|
[etcd] |
|
|
|
host = ${ETCD_ADDRESS} |
|
|
|
... |
|
|
|
|
|
|
|
Generate Client Certificate for Qinling |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
.. end |
|
|
|
|
|
|
|
See `Managing Certificates <https://kubernetes.io/docs/concepts/cluster-administration/certificates/>`_ |
|
|
|
for how to generate a client cert. We use ``cfssl`` as the example here. |
|
|
|
Generate and config client certificates for Qinling |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
#) Download and prepare the command line tools. |
|
|
|
There are a lot of |
|
|
|
`tools <https://kubernetes.io/docs/concepts/cluster-administration/certificates/>`_ |
|
|
|
out there for certificate generation. We use ``cfssl`` as the example here. |
|
|
|
|
|
|
|
#) Download and prepare the command line tools as needed. |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
@ -78,76 +103,75 @@ for how to generate a client cert. We use ``cfssl`` as the example here. |
|
|
|
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /tmp/cfssljson |
|
|
|
chmod +x /tmp/cfssljson |
|
|
|
|
|
|
|
#) Generate the client ceritificate for Qinling. Note that the common name |
|
|
|
of the subject is set to ``qinling`` in the example CSR located at |
|
|
|
``example/kubernetes/cfssl-client-csr.json``. |
|
|
|
.. end |
|
|
|
|
|
|
|
#) Generate the kubernetes and etcd client certificates for Qinling. |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
mkdir /tmp/certs; cd /tmp/certs |
|
|
|
/tmp/cfssl gencert -ca=/path/to/kubernetes_ca_crt \ |
|
|
|
-ca-key=/path/to/kubernetes_ca_key \ |
|
|
|
-config=QINLING_SOURCE_FOLDER/example/kubernetes/cfssl-ca-config.json \ |
|
|
|
mkdir -p /tmp/certs; cd /tmp/certs |
|
|
|
curl -SL https://raw.githubusercontent.com/openstack/qinling/master/example/kubernetes/cfssl-ca-config.json -o /tmp/certs/cfssl-ca-config.json |
|
|
|
curl -SL https://raw.githubusercontent.com/openstack/qinling/master/example/kubernetes/cfssl-client-csr.json -o /tmp/certs/cfssl-client-csr.json |
|
|
|
/tmp/cfssl gencert -ca=${K8S_CA_CERT} \ |
|
|
|
-ca-key=${K8S_CA_KEY} \ |
|
|
|
-config=/tmp/certs/cfssl-ca-config.json \ |
|
|
|
-profile=client \ |
|
|
|
QINLING_SOURCE_FOLDER/example/kubernetes/cfssl-client-csr.json | /tmp/cfssljson -bare client |
|
|
|
|
|
|
|
#) Copy the needed files to the locations. The command above generates two |
|
|
|
files named ``client-key.pem`` and ``client.pem``, the former is the key |
|
|
|
file of the client certificate, and the latter is the certificate file |
|
|
|
itself. |
|
|
|
|
|
|
|
.. note:: |
|
|
|
/tmp/certs/cfssl-client-csr.json | /tmp/cfssljson -bare k8s-client |
|
|
|
/tmp/cfssl gencert -ca=${ETCD_CA_CERT} \ |
|
|
|
-ca-key=${ETCD_CA_KEY} \ |
|
|
|
-config=/tmp/certs/cfssl-ca-config.json \ |
|
|
|
-profile=client \ |
|
|
|
/tmp/certs/cfssl-client-csr.json | /tmp/cfssljson -bare etcd-client |
|
|
|
|
|
|
|
Remember to backup the existing files in /etc/qinling/pki/kubernetes |
|
|
|
folder first. |
|
|
|
.. end |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
#) Move the certificates to the appropriate folders and ensure the qinling |
|
|
|
service user has permission to those folders. |
|
|
|
|
|
|
|
mkdir -p /etc/qinling/pki/kubernetes |
|
|
|
mv client-key.pem /etc/qinling/pki/kubernetes/qinling.key |
|
|
|
mv client.pem /etc/qinling/pki/kubernetes/qinling.crt |
|
|
|
mv /path/to/kubernetes_ca_crt /etc/qinling/pki/kubernetes/ca.crt |
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
.. note:: |
|
|
|
mkdir -p /etc/qinling/pki/{kubernetes,etcd} |
|
|
|
cp k8s-client-key.pem /etc/qinling/pki/kubernetes/qinling.key |
|
|
|
cp k8s-client.pem /etc/qinling/pki/kubernetes/qinling.crt |
|
|
|
cp etcd-client-key.pem /etc/qinling/pki/etcd/qinling-etcd-client.key |
|
|
|
cp etcd-client.pem /etc/qinling/pki/etcd/qinling-etcd-client.crt |
|
|
|
cp ${K8S_CA_CERT} /etc/qinling/pki/kubernetes/ca.crt |
|
|
|
cp ${ETCD_CA_CERT} /etc/qinling/pki/etcd/ca.crt |
|
|
|
chown -R qinling:qinling /etc/qinling/pki |
|
|
|
|
|
|
|
Make sure both ``/etc/qinling/pki/kubernetes`` and ``/etc/qinling/pki`` |
|
|
|
belong to Qinling service user. You can set the permissions with |
|
|
|
``chown -R qinling:qinling /etc/qinling/pki`` |
|
|
|
.. end |
|
|
|
|
|
|
|
Create Role and RoleBinding in Kubernetes |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
If RBAC Authorization is enabled, we can limit the permissions that Qinling |
|
|
|
access the Kubernetes API. Before you procceed the steps in this section, |
|
|
|
make sure that the Kubernetes API server is running with the |
|
|
|
``--authorization-mode=RBAC`` option. |
|
|
|
|
|
|
|
Qinling provides a single file located at |
|
|
|
``example/kubernetes/k8s_qinling_role.yaml`` for users to |
|
|
|
create a ``Role`` and a ``ClusterRole`` with the permissions that Qinling |
|
|
|
needs, and bind the roles to the user named ``qinling``, which is from |
|
|
|
the common name of the subject in the client certificate. The role is defined |
|
|
|
within a namespace named ``qinling``, which is the default namespace that |
|
|
|
Qinling uses and the name is configurable. |
|
|
|
According least privilege principle, the operation permission of qinling user |
|
|
|
in kubernetes cluster should be limited, this could be easily achieved by |
|
|
|
applying the pre-defined authorization manifest file. |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
curl -sSL https://raw.githubusercontent.com/openstack/qinling/master/example/kubernetes/k8s_qinling_role.yaml | kubectl apply -f - |
|
|
|
|
|
|
|
.. end |
|
|
|
|
|
|
|
Restart Qinling Engine |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
Restart qinling-engine service |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
Restart the ``qinling-engine`` service after the steps above are done, and now |
|
|
|
Qinling is accessing the Kubernetes API and etcd service using TLS. The |
|
|
|
requests that Qinling makes to the Kubernetes API are also authorized. |
|
|
|
|
|
|
|
.. code-block:: console |
|
|
|
|
|
|
|
systemctl restart devstack@qinling-engine.service |
|
|
|
|
|
|
|
Restart the qinling-engine service after the steps above are done, and now |
|
|
|
Qinling is accessing the Kubernetes API with itself authenticated by a client |
|
|
|
certificate. The requests that Qinling makes to the Kubernetes API are also |
|
|
|
authorized. |
|
|
|
.. end |
|
|
|
|
|
|
|
Access the Kubernetes API Insecurely (For testing purpose ONLY) |
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
Qinling can also connect to the Kubernetes API insecurely if the Kubernetes API |
|
|
|
server serves for insecure connections. However this is not recommended and |
|
|
|
server serves for insecure connections. However, this is not recommended and |
|
|
|
should be used for testing purpose only. |
|
|
|
|
|
|
|
In the configuration file, under the ``kubernetes`` section, set ``kube_host`` |
|
|
@ -155,3 +179,11 @@ to the URI which the Kubernetes API serves for insecure HTTP connections, for |
|
|
|
example, ``kube_host = http://localhost:8080``, and set ``use_api_certificate`` |
|
|
|
to ``False`` to disable Qinling using a client certificate to access the |
|
|
|
Kubernetes API. |
|
|
|
|
|
|
|
.. code-block:: ini |
|
|
|
|
|
|
|
[kubernetes] |
|
|
|
kube_host = http://localhost:8080 |
|
|
|
use_api_certificate = False |
|
|
|
|
|
|
|
.. end |