Fully Document node/nodelabel override capability
Explain more about how the override functionality works, and provide a full example that shows how to integrate it into a new chart being developed. Change-Id: I265a1ea9fcc501ea2c0af679fcae59e3c22d7b77
This commit is contained in:
Anderson, Craig (ca846m)
parent
d544a556db
commit
f4336b8605
@ -1,11 +1,234 @@
|
|||||||
Node and label specific configurations
|
Node and node label specific daemonset configurations
|
||||||
--------------------------------------
|
=====================================================
|
||||||
|
|
||||||
There are situations where we need to define configuration differently for
|
A typical Helm daemonset may leverage a secret to store configuration data.
|
||||||
different nodes in the environment. For example, we may require that some nodes
|
However, there are cases where the same secret document can't be used for
|
||||||
have a different vcpu_pin_set or other hardware specific deltas in nova.conf.
|
the entire daemonset, because there are node-specific differences.
|
||||||
|
|
||||||
To do this, we can specify overrides in the values fed to the chart. Ex:
|
To address this use-case, the ``helm-toolkit.utils.daemonset_overrides``
|
||||||
|
template was added in helm-toolkit. This was created with the intention that it
|
||||||
|
should be straightforward to convert (wrap) a pre-existing daemonset with the
|
||||||
|
functionality to override secret parameters on a per-node or per-nodelabel
|
||||||
|
basis.
|
||||||
|
|
||||||
|
Adapting your daemonset to support node/nodelabel overrides
|
||||||
|
-----------------------------------------------------------
|
||||||
|
|
||||||
|
Consider the following (simplified) secret and daemonset pairing example:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# Simplified secret definition
|
||||||
|
# ===============================
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
# Note ref to $secretName for dynamically generated secrets
|
||||||
|
metadata:
|
||||||
|
name: mychart-etc
|
||||||
|
data:
|
||||||
|
myConf: {{ include "helm-toolkit.utils.template" | b64enc }}
|
||||||
|
|
||||||
|
# Simplified daemonset definition
|
||||||
|
# ===============================
|
||||||
|
---
|
||||||
|
apiVersion: extensions/v1beta1
|
||||||
|
kind: DaemonSet
|
||||||
|
metadata:
|
||||||
|
name: mychart-name
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: my-container
|
||||||
|
volumes:
|
||||||
|
- name: mychart-etc
|
||||||
|
secret:
|
||||||
|
name: mychart-etc
|
||||||
|
defaultMode: 0444
|
||||||
|
|
||||||
|
Assume the chart name is ``mychart``.
|
||||||
|
|
||||||
|
Now we can wrap the existing YAML to make it support node and nodelabel
|
||||||
|
overrides, with minimal changes to the existing YAML (note where $secretName
|
||||||
|
has been substituted):
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# Simplified secret definition needed for node/nodelabel overrides
|
||||||
|
# -------------------------------------------------------------------
|
||||||
|
# Wrap secret definition
|
||||||
|
{{- define "mychart.secret.etc" }}
|
||||||
|
{{- $secretName := index . 0 }}
|
||||||
|
{{- $envAll := index . 1 }}
|
||||||
|
# Set to the same env context as was available to the caller, so we can
|
||||||
|
# access any env data needed to build the template (e.g., envAll.Values...)
|
||||||
|
{{- with $envAll }}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
# Note ref to $secretName for dynamically generated secrets
|
||||||
|
metadata:
|
||||||
|
name: {{ $secretName }}
|
||||||
|
data:
|
||||||
|
myConf: {{ include "helm-toolkit.utils.template" | b64enc }}
|
||||||
|
{{- end }}
|
||||||
|
{{- end }}
|
||||||
|
|
||||||
|
# Simplified daemonset definition needed for node/nodelabel overrides
|
||||||
|
# -------------------------------------------------------------------
|
||||||
|
# Wrap daemonset definition
|
||||||
|
{{- define "mychart.daemonset" }}
|
||||||
|
{{- $daemonset := index . 0 }}
|
||||||
|
{{- $secretName := index . 1 }}
|
||||||
|
{{- $envAll := index . 2 }}
|
||||||
|
# Set to the same env context as was available to the caller, so we can
|
||||||
|
# access any env data needed to build the template (e.g., envAll.Values...)
|
||||||
|
{{- with $envAll }}
|
||||||
|
---
|
||||||
|
apiVersion: extensions/v1beta1
|
||||||
|
kind: DaemonSet
|
||||||
|
metadata:
|
||||||
|
name: {{ $daemonset }}
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: {{ $daemonset }}
|
||||||
|
volumes:
|
||||||
|
# Note refs to $secretName for dynamically generated secrets
|
||||||
|
- name: {{ $secretName }}
|
||||||
|
secret:
|
||||||
|
name: {{ $secretName }}
|
||||||
|
defaultMode: 0444
|
||||||
|
{{- end }}
|
||||||
|
{{- end }}
|
||||||
|
# Desired daemonset name/prefix that helm will register with kubernetes
|
||||||
|
# Note that this needs to be a valid dns-1123 name for a k8s resource
|
||||||
|
{{- $daemonset := "mydaemonset" }}
|
||||||
|
# Desired secret name/prefix that helm will register with kubernetes
|
||||||
|
# Note that this needs to be a valid dns-1123 name for a k8s resource
|
||||||
|
{{- $secretName := "mychart-etc" }}
|
||||||
|
# Generate the daemonset YAML with a matching/consistent secretName (so
|
||||||
|
# daemonset_overrides knows which volumes to dynamically substitute with the
|
||||||
|
# auto-generated secrets). You may include in this list any other vars
|
||||||
|
# which you need to reference or substitute into the daemonset YAML above.
|
||||||
|
{{- $daemonset_yaml := list $secretName . | include "mychart.daemonset" | toString | fromYaml }}
|
||||||
|
# Namespace to the secret definition which will be used/manipulated
|
||||||
|
{{- $secret_include := "mychart.secret.etc" }}
|
||||||
|
# Pass all these elements to daemonset_overrides to generate secret/daemonset
|
||||||
|
# pairings for each set of overrides (plus one with no overrides)
|
||||||
|
{{- list $daemonset $daemonset_yaml $secret_include $secretName . | include "helm-toolkit.utils.daemonset_overrides" }}
|
||||||
|
|
||||||
|
Your daemonset should now support node and nodelabl level overrides. (Note that
|
||||||
|
you will also need your chart to have helm-toolkit listed as a dependency.)
|
||||||
|
|
||||||
|
Implementation details of node/nodelabel overrides
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
Instead of having one daemonset with one monolithic secret, this helm-toolkit
|
||||||
|
feature permits a common daemonset and secret template, from which daemonset
|
||||||
|
and secret pairings are auto-generated. It supports establishing value
|
||||||
|
overrides for nodes with specific label value pairs and for targeting nodes with
|
||||||
|
specific hostnames and hostlabels. The overridden configuration is merged with
|
||||||
|
the normal config data, with the override data taking precedence.
|
||||||
|
|
||||||
|
The chart will then generate one daemonset for each host and label override, in
|
||||||
|
addition to a default daemonset for which no overrides are applied. Each
|
||||||
|
daemonset generated will also exclude from its scheduling criteria all other
|
||||||
|
hosts and labels defined in other overrides for the same daemonset, to ensure
|
||||||
|
that there is no overlap of daemonsets (i.e., one and only one daemonset of a
|
||||||
|
given type for each node).
|
||||||
|
|
||||||
|
For example, if you have some special conf setting that should be applied
|
||||||
|
to ``host1.fqdn``, and another special conf setting that should be applied
|
||||||
|
to nodes labeled with ``someNodeLabel``, then three secret/daemonset pairs
|
||||||
|
will be generated and registered with kubernetes: one for ``host1.fqdn``, one
|
||||||
|
for ``someNodeLabel``, and one for ``default``.
|
||||||
|
|
||||||
|
The order of precedence for matches is FQDN, node label, and then default. If a
|
||||||
|
node matches both a FQDN and a nodelabel, then only the FQDN override is applied.
|
||||||
|
Pay special attention to adding FQDN overrides for nodes that match a nodelabel
|
||||||
|
override, as you would need to duplicate the nodelabel overrides for that node
|
||||||
|
in the FQDN overrides for them to still apply.
|
||||||
|
|
||||||
|
If there is no matching FQDN and no matching nodelabel, then the default
|
||||||
|
daemonset/secret (with no overrides applied) is used.
|
||||||
|
|
||||||
|
If a node matches more than one nodelabel, only the last matching nodelabel will
|
||||||
|
apply (last in terms of the order the overrides are defined in the YAML).
|
||||||
|
|
||||||
|
Exercising node/nodelabel overrides
|
||||||
|
-----------------------------------
|
||||||
|
|
||||||
|
The following example demonstrates how to exercise the node/nodelabel overrides:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
data:
|
||||||
|
values:
|
||||||
|
conf:
|
||||||
|
mychart:
|
||||||
|
foo: 1
|
||||||
|
# "overrides" keyword to invoke override behavior
|
||||||
|
overrides:
|
||||||
|
# To match these overrides to the right daemonset, the following key
|
||||||
|
# needs to follow the pattern:
|
||||||
|
# Chart.Name + '_' + $daemonset
|
||||||
|
# where $daemonset is the value set for $daemonset in the daemonset
|
||||||
|
# config above.
|
||||||
|
mychart_mydaemonset:
|
||||||
|
# labels dict contains a list of labels which overrides apply to. Dict may be excluded
|
||||||
|
# if there are no labels to override.
|
||||||
|
# Note - if a host satisfies more than one label in this list, then whichever matching
|
||||||
|
# label is furtherest down on the list will be the one applied to the node. E.g., if
|
||||||
|
# a host matched both label criteria below, then the overrides for "another_label"
|
||||||
|
# would be applied.
|
||||||
|
labels:
|
||||||
|
# node label key and values to match against to apply these config overrides.
|
||||||
|
# The values are ORed, so the daemonset will spawn to all nodes to node_type
|
||||||
|
# set to "foo" and to all nodes with node_type set to "bar".
|
||||||
|
- label:
|
||||||
|
key: node_type
|
||||||
|
values:
|
||||||
|
- "foo"
|
||||||
|
- "bar"
|
||||||
|
# The setting overrides that will be applied for hosts with this host label
|
||||||
|
conf:
|
||||||
|
mychart:
|
||||||
|
foo: 2
|
||||||
|
# another label/key to match against to apply different overrides
|
||||||
|
- label:
|
||||||
|
key: another_label
|
||||||
|
values:
|
||||||
|
- "another_value"
|
||||||
|
# The setting overrides that will be applied for hosts with this host label
|
||||||
|
conf:
|
||||||
|
mychart:
|
||||||
|
foo: 3
|
||||||
|
# hosts dict contains a list of hosts which overrides apply to. Dict may be excluded
|
||||||
|
# if there are no hosts to override.
|
||||||
|
hosts:
|
||||||
|
# FQDN of the host to override settings on
|
||||||
|
- name: superhost
|
||||||
|
# The setting overrides that will be applied for this host
|
||||||
|
conf:
|
||||||
|
mychart:
|
||||||
|
foo: 4
|
||||||
|
# FQDN of another host to override settings on
|
||||||
|
- name: superhost2
|
||||||
|
# The setting overrides that will be applied for this host
|
||||||
|
conf:
|
||||||
|
mychart:
|
||||||
|
foo: 5
|
||||||
|
|
||||||
|
Nova vcpu example
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Some nodes may have a different vcpu_pin_set in nova.conf due to differences
|
||||||
|
in CPU hardware.
|
||||||
|
|
||||||
|
To address this, we can specify overrides in the values fed to the chart. Ex:
|
||||||
|
|
||||||
.. code-block:: yaml
|
.. code-block:: yaml
|
||||||
|
|
||||||
@ -53,7 +276,7 @@ Note that only one set of overrides is applied per node, such that:
|
|||||||
e.g., "another-label" overrides will apply to a node containing both labels.
|
e.g., "another-label" overrides will apply to a node containing both labels.
|
||||||
|
|
||||||
Also note that other non-overridden values are inherited by hosts and labels with overrides.
|
Also note that other non-overridden values are inherited by hosts and labels with overrides.
|
||||||
The following shows a set of example hosts and the values fed into the configmap for each:
|
The following shows a set of example hosts and the values fed into each:
|
||||||
|
|
||||||
1. ``host1.fqdn`` with labels ``compute-type: dpdk, sriov`` and ``another-label: another-value``:
|
1. ``host1.fqdn`` with labels ``compute-type: dpdk, sriov`` and ``another-label: another-value``:
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user