e965208807
removed trailing white spaces added comments from Patchset 2 Updated example output to fix indentations Updated kubernetes-custom-configuration-31c1fd41857d.rst Signed-off-by: Oliver Loten <Oliver.Loten@windriver.com> Change-Id: Idbc0f9a03fbcc67f075367cdfba18a3053228a95
362 lines
12 KiB
ReStructuredText
362 lines
12 KiB
ReStructuredText
.. _kubernetes-custom-configuration-31c1fd41857d:
|
|
|
|
===============================
|
|
Kubernetes Custom Configuration
|
|
===============================
|
|
|
|
------------
|
|
Introduction
|
|
------------
|
|
|
|
Kubernetes configuration can be customized during deployment by specifying
|
|
bootstrap overrides in the localhost.yml file during the Ansible bootstrap
|
|
process.
|
|
|
|
.. note::
|
|
|
|
Kubernetes custom configuration is only applicable to the bootstrap phase
|
|
and runtime support is not available.
|
|
|
|
Custom configuration includes:
|
|
|
|
- Configuring options on kube-apiserver such as feature gates and admission
|
|
controllers,
|
|
|
|
- Configuring options on kube-controller-manager such as node-monitor-period
|
|
and pod-eviction-timeout,
|
|
|
|
- Configuring options on kube-scheduler such as feature gates
|
|
|
|
- Configuring options on kubelet such as maximum pods and enabling unsafe
|
|
sysctls.
|
|
|
|
----------------------------
|
|
kube-apiserver configuration
|
|
----------------------------
|
|
|
|
The Kubernetes API server validates and configures data for the api objects
|
|
which include pods, services, replicationcontrollers, and others. The API
|
|
Server services REST operations and provides the frontend to the cluster's
|
|
shared state through which all other components interact.
|
|
|
|
For a list of all configurable options of kube-apiserver, see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ .
|
|
|
|
To set or override a kube-apiserver option, add the desired parameters to an
|
|
apiserver_extra_args section in the localhost.yml.
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
apiserver_extra_args:
|
|
admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
|
|
audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
|
|
default-not-ready-toleration-seconds: "35"
|
|
default-unreachable-toleration-seconds: "35"
|
|
feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
|
|
enable-admission-plugins: "NodeRestriction,PodNodeSelector"
|
|
event-ttl: "20h"
|
|
|
|
-------------------------------------
|
|
kube-controller-manager configuration
|
|
-------------------------------------
|
|
|
|
The Kubernetes controller manager is a daemon that embeds the core control
|
|
loops shipped with Kubernetes. A controller is a control loop that watches the
|
|
shared state of the cluster through the apiserver and makes changes attempting
|
|
to move the current state towards the desired state.
|
|
|
|
For a list of all configurable options of kube-controller-manager, see
|
|
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
|
|
.
|
|
|
|
To set or override a kube-controller-manager option, add the desired parameters
|
|
to an controllermanager_extra_args section in the localhost.yml .
|
|
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
controllermanager_extra_args:
|
|
node-monitor-period: "4s"
|
|
node-monitor-grace-period: "25s"
|
|
pod-eviction-timeout: '35s'
|
|
feature-gates: "TTLAfterFinished=true,MemoryManager=true"
|
|
|
|
----------------------------
|
|
kube-scheduler configuration
|
|
----------------------------
|
|
|
|
The Kubernetes scheduler is a control plane process which assigns Pods to
|
|
Nodes. The scheduler determines which Nodes are valid placements for each Pod
|
|
in the scheduling queue according to constraints and available resources. The
|
|
scheduler then ranks each valid Node and binds the Pod to a suitable Node.
|
|
Multiple different schedulers may be used within a cluster; kube-scheduler is
|
|
the reference implementation.
|
|
|
|
For a list of all configurable options of kube-scheduler, see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/> .
|
|
|
|
To set or override a kube-scheduler option, add the desired parameters to an
|
|
scheduler_extra_args section in the localhost.yml .
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
scheduler_extra_args:
|
|
feature-gates: "TTLAfterFinished=false"
|
|
|
|
----------------------
|
|
kubelet configurations
|
|
----------------------
|
|
|
|
The kubelet is the primary "node agent" that runs on each node.
|
|
|
|
For a list of all configurable options, see https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ .
|
|
|
|
To set or override a kubelet option, add the desired parameters to an
|
|
kubelet_configurations section in the localhost.yml .
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
kubelet_configurations:
|
|
featureGates:
|
|
MemoryManager: true
|
|
HugePageStorageMediumSize: true
|
|
|
|
---------------------------------------------------------------------------------
|
|
apiserver_extra_volumes, controllermanager_extra_volumes, scheduler_extra_volumes
|
|
---------------------------------------------------------------------------------
|
|
|
|
Some options/parameters specified in **apiserver_extra_args**,
|
|
**controllermanager_extra_args** and **scheduler_extra_volumes** refer to a
|
|
configuration file. The contents of these files are configured in the
|
|
bootstrap overrides (localhost.yml) with the apiserver_extra_volumes,
|
|
controllermanager_extra_volumes and scheduler_extra_volumes definitions.
|
|
|
|
For instance, if admission plugins are configured and need additional
|
|
configuration, that configuration should be set in a specific file referenced
|
|
by the **admission-control-config-file** parameter.
|
|
|
|
See the example below where the 'admission-control-config-file' option and the
|
|
'PodNodeSelector' admission plugin is specified for kube-apiserver. Both of
|
|
these options require the specification of a yaml file.
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
apiserver_extra_args:
|
|
admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yaml"
|
|
enable-admission-plugins: "PodNodeSelector"
|
|
|
|
apiserver_extra_volumes:
|
|
- name: admission-control-config-file
|
|
mountPath: "/etc/kubernetes/admission-control-config-file.yaml"
|
|
readOnly: true
|
|
pathType: "File"
|
|
content: |
|
|
apiVersion: apiserver.config.k8s.io/v1
|
|
kind: AdmissionConfiguration
|
|
plugins:
|
|
- name: PodSecurity
|
|
configuration:
|
|
apiVersion: pod-security.admission.config.k8s.io/v1beta1
|
|
kind: PodSecurityConfiguration
|
|
# Defaults applied when a mode label is not set.
|
|
#
|
|
# Level label values must be one of:
|
|
# - "privileged" (default)
|
|
# - "baseline"
|
|
# - "restricted"
|
|
#
|
|
# Version label values must be one of:
|
|
# - "latest" (default)
|
|
# - specific version like "v1.24"
|
|
defaults:
|
|
enforce: "privileged"
|
|
enforce-version: "latest"
|
|
audit: "privileged"
|
|
audit-version: "latest"
|
|
warn: "privileged"
|
|
warn-version: "latest"
|
|
exemptions:
|
|
# Array of authenticated usernames to exempt.
|
|
usernames: []
|
|
# Array of runtime class names to exempt.
|
|
runtimeClasses: []
|
|
# Array of namespaces to exempt.
|
|
namespaces: []
|
|
- name: pod-node-selector
|
|
mountPath: "/etc/kubernetes/pod-node-selector.yaml"
|
|
readOnly: true
|
|
pathType: "File"
|
|
content: |
|
|
podNodeSelectorPluginConfig:
|
|
clusterDefaultNodeSelector: name-of-node-selector
|
|
namespace1: name-of-node-selector
|
|
namespace2: name-of-node-selector
|
|
|
|
The example below enables kubernetes auditing which requires an
|
|
audit-policy-file.yaml file to specify the details of what events should be
|
|
audited.
|
|
|
|
Example usage:
|
|
|
|
.. code-block::
|
|
|
|
|
|
apiserver_extra_args:
|
|
audit-policy-file: /etc/kubernetes/audit-policy-file.yaml
|
|
audit-log-path: /var/log/kubernetes/audit/audit.log
|
|
|
|
apiserver_extra_volumes:
|
|
- name: audit-policy-file
|
|
mountPath: "/etc/kubernetes/audit-policy-file.yaml"
|
|
readOnly: true
|
|
pathType: "File"
|
|
content: |
|
|
# Log all requests at the Metadata level.
|
|
apiVersion: audit.k8s.io/v1
|
|
kind: Policy
|
|
rules:
|
|
- level: Metadata
|
|
- name: audit-log-path
|
|
mountPath: "/var/log/kubernetes/audit/"
|
|
readOnly: false
|
|
pathType: "DirectoryOrCreate"
|
|
|
|
-----------------------------
|
|
Complex Example configuration
|
|
-----------------------------
|
|
|
|
.. code-block::
|
|
|
|
apiserver_extra_args:
|
|
admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
|
|
audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
|
|
default-not-ready-toleration-seconds: "35"
|
|
default-unreachable-toleration-seconds: "35"
|
|
feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
|
|
enable-admission-plugins: "NodeRestriction,PodNodeSelector"
|
|
event-ttl: "20h"
|
|
audit-log-path: "/var/log/kubernetes/audit/audit.log"
|
|
audit-log-maxage: "1"
|
|
audit-log-maxbackup: "2"
|
|
audit-log-maxsize: "1"
|
|
|
|
scheduler_extra_args:
|
|
feature-gates: "TTLAfterFinished=false"
|
|
|
|
controllermanager_extra_args:
|
|
node-monitor-period: "4s"
|
|
node-monitor-grace-period: "25s"
|
|
pod-eviction-timeout: '35s'
|
|
feature-gates: "TTLAfterFinished=true,MemoryManager=true"
|
|
|
|
kubelet_configurations:
|
|
featureGates:
|
|
MemoryManager: true
|
|
HugePageStorageMediumSize: true
|
|
|
|
apiserver_extra_volumes:
|
|
- name: admission-control-config-file
|
|
mountPath: "/etc/kubernetes/admission-control-config-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
apiVersion: apiserver.config.k8s.io/v1
|
|
kind: AdmissionConfiguration
|
|
plugins:
|
|
- name: PodNodeSelector
|
|
path: /etc/kubernetes/podnodeselector.yaml
|
|
- name: pod-nodes-selector-plugin-config
|
|
mountPath: "/etc/kubernetes/podnodeselector.yaml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
podNodeSelecto+rPluginConfig:
|
|
clusterDefaultNodeSelector: name-of-node-selector
|
|
namespace1: name-of-node-selector
|
|
namespace2: name-of-node-selector
|
|
- name: audit-policy-file
|
|
mountPath: "/etc/kubernetes/audit-policy-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
# Log all requests at the Metadata level.
|
|
apiVersion: audit.k8s.io/v1
|
|
kind: Policy
|
|
rules:
|
|
- level: Metadata
|
|
- name: audit-log-path
|
|
mountPath: "/var/log/kubernetes/audit/"
|
|
readOnly: false
|
|
pathType: 'DirectoryOrCreate'
|
|
|
|
scheduler_extra_volumes:
|
|
- name: sch-admission-control-config-file
|
|
mountPath: "/etc/kubernetes/admission-control-config-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
apiVersion:
|
|
kind: AdmissionConfiguration
|
|
plugins:
|
|
- name: PodNodeSelector
|
|
path: /etc/kubernetes/podnodeselector.yaml
|
|
- name: sch-pod-nodes-selector-plugin-config
|
|
mountPath: "/etc/kubernetes/podnodeselector.yaml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
podNodeSelectorPluginConfig:
|
|
clusterDefaultNodeSelector: name-of-node-selector
|
|
namespace1: name-of-node-selector
|
|
namespace2: name-of-node-selector
|
|
- name: sch-audit-policy-file
|
|
mountPath: "/etc/kubernetes/audit-policy-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
# Log all requests at the Metadata level.
|
|
apiVersion: audit.k8s.io/v1
|
|
kind: Policy
|
|
rules:
|
|
- level: Metadata
|
|
|
|
controllermanager_extra_volumes:
|
|
- name: cm-admission-control-config-file
|
|
mountPath: "/etc/kubernetes/admission-control-config-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
apiVersion: apiserver.config.k8s.io/v1
|
|
kind: AdmissionConfiguration
|
|
plugins:
|
|
- name: PodNodeSelector
|
|
path: /etc/kubernetes/podnodeselector.yaml
|
|
- name: cm-pod-nodes-selector-plugin-config
|
|
mountPath: "/etc/kubernetes/podnodeselector.yaml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
podNodeSelectorPluginConfig:
|
|
clusterDefaultNodeSelector: name-of-node-selector
|
|
namespace1: name-of-node-selector
|
|
namespace2: name-of-node-selector
|
|
- name: cm-audit-policy-file
|
|
mountPath: "/etc/kubernetes/audit-policy-file.yml"
|
|
pathType: "File"
|
|
readOnly: true
|
|
content: |
|
|
# Log all requests at the Metadata level.
|
|
apiVersion: audit.k8s.io/v1
|
|
kind: Policy
|
|
rules:
|
|
- level: Metadata
|
|
|