diff --git a/doc/source/devref/node-and-label-specific-configurations.rst b/doc/source/devref/node-and-label-specific-configurations.rst index 41ed9c901e..6b32864b52 100644 --- a/doc/source/devref/node-and-label-specific-configurations.rst +++ b/doc/source/devref/node-and-label-specific-configurations.rst @@ -1,11 +1,234 @@ -Node and label specific configurations --------------------------------------- +Node and node label specific daemonset configurations +===================================================== -There are situations where we need to define configuration differently for -different nodes in the environment. For example, we may require that some nodes -have a different vcpu_pin_set or other hardware specific deltas in nova.conf. +A typical Helm daemonset may leverage a secret to store configuration data. +However, there are cases where the same secret document can't be used for +the entire daemonset, because there are node-specific differences. -To do this, we can specify overrides in the values fed to the chart. Ex: +To address this use-case, the ``helm-toolkit.utils.daemonset_overrides`` +template was added in helm-toolkit. This was created with the intention that it +should be straightforward to convert (wrap) a pre-existing daemonset with the +functionality to override secret parameters on a per-node or per-nodelabel +basis. + +Adapting your daemonset to support node/nodelabel overrides +----------------------------------------------------------- + +Consider the following (simplified) secret and daemonset pairing example: + +.. code-block:: yaml + + # Simplified secret definition + # =============================== + --- + apiVersion: v1 + kind: Secret + # Note ref to $secretName for dynamically generated secrets + metadata: + name: mychart-etc + data: + myConf: {{ include "helm-toolkit.utils.template" | b64enc }} + + # Simplified daemonset definition + # =============================== + --- + apiVersion: extensions/v1beta1 + kind: DaemonSet + metadata: + name: mychart-name + spec: + template: + spec: + containers: + - name: my-container + volumes: + - name: mychart-etc + secret: + name: mychart-etc + defaultMode: 0444 + +Assume the chart name is ``mychart``. + +Now we can wrap the existing YAML to make it support node and nodelabel +overrides, with minimal changes to the existing YAML (note where $secretName +has been substituted): + +.. code-block:: yaml + + # Simplified secret definition needed for node/nodelabel overrides + # ------------------------------------------------------------------- + # Wrap secret definition + {{- define "mychart.secret.etc" }} + {{- $secretName := index . 0 }} + {{- $envAll := index . 1 }} + # Set to the same env context as was available to the caller, so we can + # access any env data needed to build the template (e.g., envAll.Values...) + {{- with $envAll }} + --- + apiVersion: v1 + kind: Secret + # Note ref to $secretName for dynamically generated secrets + metadata: + name: {{ $secretName }} + data: + myConf: {{ include "helm-toolkit.utils.template" | b64enc }} + {{- end }} + {{- end }} + + # Simplified daemonset definition needed for node/nodelabel overrides + # ------------------------------------------------------------------- + # Wrap daemonset definition + {{- define "mychart.daemonset" }} + {{- $daemonset := index . 0 }} + {{- $secretName := index . 1 }} + {{- $envAll := index . 2 }} + # Set to the same env context as was available to the caller, so we can + # access any env data needed to build the template (e.g., envAll.Values...) + {{- with $envAll }} + --- + apiVersion: extensions/v1beta1 + kind: DaemonSet + metadata: + name: {{ $daemonset }} + spec: + template: + spec: + containers: + - name: {{ $daemonset }} + volumes: + # Note refs to $secretName for dynamically generated secrets + - name: {{ $secretName }} + secret: + name: {{ $secretName }} + defaultMode: 0444 + {{- end }} + {{- end }} + # Desired daemonset name/prefix that helm will register with kubernetes + # Note that this needs to be a valid dns-1123 name for a k8s resource + {{- $daemonset := "mydaemonset" }} + # Desired secret name/prefix that helm will register with kubernetes + # Note that this needs to be a valid dns-1123 name for a k8s resource + {{- $secretName := "mychart-etc" }} + # Generate the daemonset YAML with a matching/consistent secretName (so + # daemonset_overrides knows which volumes to dynamically substitute with the + # auto-generated secrets). You may include in this list any other vars + # which you need to reference or substitute into the daemonset YAML above. + {{- $daemonset_yaml := list $secretName . | include "mychart.daemonset" | toString | fromYaml }} + # Namespace to the secret definition which will be used/manipulated + {{- $secret_include := "mychart.secret.etc" }} + # Pass all these elements to daemonset_overrides to generate secret/daemonset + # pairings for each set of overrides (plus one with no overrides) + {{- list $daemonset $daemonset_yaml $secret_include $secretName . | include "helm-toolkit.utils.daemonset_overrides" }} + +Your daemonset should now support node and nodelabl level overrides. (Note that +you will also need your chart to have helm-toolkit listed as a dependency.) + +Implementation details of node/nodelabel overrides +-------------------------------------------------- + +Instead of having one daemonset with one monolithic secret, this helm-toolkit +feature permits a common daemonset and secret template, from which daemonset +and secret pairings are auto-generated. It supports establishing value +overrides for nodes with specific label value pairs and for targeting nodes with +specific hostnames and hostlabels. The overridden configuration is merged with +the normal config data, with the override data taking precedence. + +The chart will then generate one daemonset for each host and label override, in +addition to a default daemonset for which no overrides are applied. Each +daemonset generated will also exclude from its scheduling criteria all other +hosts and labels defined in other overrides for the same daemonset, to ensure +that there is no overlap of daemonsets (i.e., one and only one daemonset of a +given type for each node). + +For example, if you have some special conf setting that should be applied +to ``host1.fqdn``, and another special conf setting that should be applied +to nodes labeled with ``someNodeLabel``, then three secret/daemonset pairs +will be generated and registered with kubernetes: one for ``host1.fqdn``, one +for ``someNodeLabel``, and one for ``default``. + +The order of precedence for matches is FQDN, node label, and then default. If a +node matches both a FQDN and a nodelabel, then only the FQDN override is applied. +Pay special attention to adding FQDN overrides for nodes that match a nodelabel +override, as you would need to duplicate the nodelabel overrides for that node +in the FQDN overrides for them to still apply. + +If there is no matching FQDN and no matching nodelabel, then the default +daemonset/secret (with no overrides applied) is used. + +If a node matches more than one nodelabel, only the last matching nodelabel will +apply (last in terms of the order the overrides are defined in the YAML). + +Exercising node/nodelabel overrides +----------------------------------- + +The following example demonstrates how to exercise the node/nodelabel overrides: + +.. code-block:: yaml + + data: + values: + conf: + mychart: + foo: 1 + # "overrides" keyword to invoke override behavior + overrides: + # To match these overrides to the right daemonset, the following key + # needs to follow the pattern: + # Chart.Name + '_' + $daemonset + # where $daemonset is the value set for $daemonset in the daemonset + # config above. + mychart_mydaemonset: + # labels dict contains a list of labels which overrides apply to. Dict may be excluded + # if there are no labels to override. + # Note - if a host satisfies more than one label in this list, then whichever matching + # label is furtherest down on the list will be the one applied to the node. E.g., if + # a host matched both label criteria below, then the overrides for "another_label" + # would be applied. + labels: + # node label key and values to match against to apply these config overrides. + # The values are ORed, so the daemonset will spawn to all nodes to node_type + # set to "foo" and to all nodes with node_type set to "bar". + - label: + key: node_type + values: + - "foo" + - "bar" + # The setting overrides that will be applied for hosts with this host label + conf: + mychart: + foo: 2 + # another label/key to match against to apply different overrides + - label: + key: another_label + values: + - "another_value" + # The setting overrides that will be applied for hosts with this host label + conf: + mychart: + foo: 3 + # hosts dict contains a list of hosts which overrides apply to. Dict may be excluded + # if there are no hosts to override. + hosts: + # FQDN of the host to override settings on + - name: superhost + # The setting overrides that will be applied for this host + conf: + mychart: + foo: 4 + # FQDN of another host to override settings on + - name: superhost2 + # The setting overrides that will be applied for this host + conf: + mychart: + foo: 5 + +Nova vcpu example +------------------ + +Some nodes may have a different vcpu_pin_set in nova.conf due to differences +in CPU hardware. + +To address this, we can specify overrides in the values fed to the chart. Ex: .. code-block:: yaml @@ -53,7 +276,7 @@ Note that only one set of overrides is applied per node, such that: e.g., "another-label" overrides will apply to a node containing both labels. Also note that other non-overridden values are inherited by hosts and labels with overrides. -The following shows a set of example hosts and the values fed into the configmap for each: +The following shows a set of example hosts and the values fed into each: 1. ``host1.fqdn`` with labels ``compute-type: dpdk, sriov`` and ``another-label: another-value``: