airship/airshipctl
Code Issues Proposed changes

Add an m3 host generator function

Browse Source 
This PS has a function which constructs a collection of Metal3 BareMetalHost
resources, along with associated configuration Secrets.
It solves for a couple of things:

1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values

This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.

More details can be found in the README.md in this patchset.

Closes: #245

Change-Id: I3ddbd36dc53ea6afbd633098c985f4b28bcbb793
This commit is contained in:
Matt McEuen 2020-06-10 18:41:39 -05:00
parent 1d34132b87
commit 0324993d60
29 changed files with 370 additions and 134 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
manifests/function/hostgenerator-m3/README.md Normal file
View File

@ -0,0 +1,37 @@
Function: hostgenerator-m3
==========================
This function constructs a collection of Metal3 BareMetalHost resources,
along with associated configuration Secrets. It solves for a couple of things:
1. pulling the nitty gritty details for generating BMH into one reusable place,
2. allowing the site-specific details to be filled in via catalogues of values
This function leverages a couple of different plugins in sequence:
The airshipctl Replacement plugin, which pulls the site-specific data from
the catalogue documents into a Templater plugin configuration; and then
the airshipctl Templater plugin, which generates a variable number of
BMHs in a data-driven fashion.
To use this function, do the following:
* Supply a `common-networking-catalogue`, which outlines things that are
typically common across hosts in a site, such as networking interfaces,
DNS servers, and other networking info.
Example: `manifests/type/gating/shared/catalogues/common-networking.yaml`
* Supply a `host-catalogue`, which contains host-specific data, such as
IP addresses and BMC information.
Example: `manifests/site/test-site/shared/catalogues/hosts.yaml`
* Supply a `host-generation-catalogue` for each `phase` that needs to
deploy one or more BMHs. This catalogue simply lists the specific
hosts that should be deployed during that phase.
Example: `manifests/site/test-site/ephemeral/bootstrap/hostgenerator/host-generation.yaml`
* If any per-host changes need to be made, they can be layered on top as
site- or phase-specific Kustomize patches against the generated
documents. E.g, if one host has a different network interface name,
or if different details need to be used during ISO bootstrapping
and normal deployment.
Example: `manifests/site/test-site/ephemeral/bootstrap/baremetalhost.yaml`

69
manifests/function/hostgenerator-m3/hosttemplate.yaml Normal file
View File

@ -0,0 +1,69 @@
apiVersion: airshipit.org/v1alpha1
kind: Templater
metadata:
name: m3-host-template
# values:
# hosts:
# (filled in from the comprehensive site-wide host-catalogue)
# hostsToGenerate:
# (filled in with phase-specific host-generation-catalogue)
# commonNetworking:
# (filled in with the type-specific common-networking-catalogue)
template: |
{{- $envAll := . }}
{{- range .hostsToGenerate }}
{{- $hostName := . }}
{{- $host := index $envAll.hosts $hostName }}
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
annotations:
labels:
name: {{ $hostName }}
spec:
online: false
bootMACAddress: {{ $host.macAddress }}
networkData:
name: {{ $hostName }}-network-data
namespace: default
bmc:
address: {{ $host.bmcAddress }}
credentialsName: {{ $hostName }}-bmc-secret
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $hostName }}-bmc-secret
data:
username: {{ $host.bmcUsername | b64enc }}
password: {{ $host.bmcPassword | b64enc }}
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $hostName }}-network-data
stringData:
networkData: |
links:
{{- range $envAll.commonNetworking.links }}
-
{{ toYaml . | indent 6 }}
{{- if $host.macAddresses }}
ethernet_mac_address: {{ index $host.macAddresses .id }}
{{- end }}
{{- end }}
networks:
{{- range $envAll.commonNetworking.networks }}
-
{{ toYaml . | indent 6 }}
ip_address: {{ index $host.ipAddresses .id }}
{{- end }}
services:
{{ toYaml $envAll.commonNetworking.services | indent 6 }}
type: Opaque
{{ end -}}

4
manifests/function/hostgenerator-m3/kustomization.yaml Normal file
View File

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- hosttemplate.yaml

34
manifests/function/hostgenerator-m3/replacements/hosts.yaml Normal file
View File

@ -0,0 +1,34 @@
# These rules inject host-specific information from the `host-catalogue`
# into the hostgenerator-m3 function's Template plugin config.
apiVersion: airshipit.org/v1alpha1
kind: ReplacementTransformer
metadata:
name: m3-host-replacements
replacements:
- source:
objref:
name: host-catalogue
fieldref: hosts.m3
target:
objref:
kind: Templater
name: m3-host-template
fieldrefs: [values.hosts]
- source:
objref:
name: host-generation-catalogue
fieldref: hosts.m3
target:
objref:
kind: Templater
name: m3-host-template
fieldrefs: [values.hostsToGenerate]
- source:
objref:
name: common-networking-catalogue
fieldref: commonNetworking
target:
objref:
kind: Templater
name: m3-host-template
fieldrefs: [values.commonNetworking]

4
manifests/function/hostgenerator-m3/replacements/kustomization.yaml Normal file
View File

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- hosts.yaml

6
manifests/site/test-site/shared/baremetalhost/node02/baremetalhost.yaml → manifests/site/test-site/ephemeral/bootstrap/baremetalhost.yaml
View File

@ -1,3 +1,4 @@
# This patches the node02 BMH to be suitable for ephemeral purposes
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
@ -8,13 +9,8 @@ metadata:
name: node02
spec:
online: true
bootMACAddress: 00:3b:8b:0c:ec:8b
networkData:
name: node02-network-data
namespace: default
bmc:
address: redfish+https://localhost:8443/redfish/v1/Systems/air-ephemeral
credentialsName: node02-bmc-secret
status:
provisioning:
# we need this status to make sure, that the host is not going to be

10
manifests/site/test-site/ephemeral/bootstrap/hostgenerator/host-generation.yaml Normal file
View File

@ -0,0 +1,10 @@
# Site-level, phase-specific lists of hosts to generate
# This is used by the hostgenerator-m3 function to narrow down the site-level
# host-catalogue to just the hosts needed for a particular phase.
apiVersion: airshipit.org/v1alpha1
kind: VariableCatalogue
metadata:
name: host-generation-catalogue
hosts:
m3:
- node02

9
manifests/site/test-site/ephemeral/bootstrap/hostgenerator/kustomization.yaml Normal file
View File

@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../../../function/hostgenerator-m3
- ../../../shared/catalogues/
- host-generation.yaml
transformers:
- ../../../../../function/hostgenerator-m3/replacements

7
manifests/site/test-site/ephemeral/bootstrap/kustomization.yaml
View File

@ -1,5 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../shared/baremetalhost/node02
- ../../../../type/gating
generators:
- hostgenerator
patchesStrategicMerge:
- baremetalhost.yaml

10
manifests/site/test-site/ephemeral/controlplane/hostgenerator/host-generation.yaml Normal file
View File

@ -0,0 +1,10 @@
# Site-level, phase-specific lists of hosts to generate
# This is used by the hostgenerator-m3 function to narrow down the site-level
# host-catalogue to just the hosts needed for a particular phase.
apiVersion: airshipit.org/v1alpha1
kind: VariableCatalogue
metadata:
name: host-generation-catalogue
hosts:
m3:
- node01

9
manifests/site/test-site/ephemeral/controlplane/hostgenerator/kustomization.yaml Normal file
View File

@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../../../function/hostgenerator-m3
- ../../../shared/catalogues/
- host-generation.yaml
transformers:
- ../../../../../function/hostgenerator-m3/replacements

6
manifests/site/test-site/ephemeral/controlplane/nodes/kustomization.yaml
View File

@ -1,6 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../shared/baremetalhost/node01
generators:
- ../hostgenerator
commonLabels:
airshipit.org/k8s-role: controlplane-host

14
manifests/site/test-site/shared/baremetalhost/node01/baremetalhost.yaml
View File

@ -1,14 +0,0 @@
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: node01
spec:
bmc:
address: redfish+http://10.23.25.1:8000/redfish/v1/Systems/air-target-1
credentialsName: node01-bmc
online: false
bootMACAddress: 52:54:00:b6:ed:31
networkData:
name: node01-netdata
namespace: default

16
manifests/site/test-site/shared/baremetalhost/node01/kustomization.yaml
View File

@ -1,16 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- baremetalhost.yaml
generatorOptions:
disableNameSuffixHash: true
secretGenerator:
- name: node01-netdata
files:
- networkData
- name: node01-bmc
literals:
- username=root
- password=r00tme

31
manifests/site/test-site/shared/baremetalhost/node01/networkData
View File

@ -1,31 +0,0 @@
links:
- id: oam0
name: oam0
type: phy
ethernet_mac_address: 52:54:00:9b:27:4c
mtu: "1500"
- id: pxe0
name: pxe0
type: phy
ethernet_mac_address: 52:54:00:b6:ed:31
mtu: "1500"
networks:
- id: private-ipv4
type: ipv4
link: oam0
ip_address: 10.23.25.102
netmask: 255.255.255.0
routes:
- network: 0.0.0.0
netmask: 0.0.0.0
gateway: 10.23.25.1
- id: private-ipv4
type: ipv4
link: pxe0
ip_address: 10.23.24.102
netmask: 255.255.255.0
services:
- address: 8.8.8.8
type: dns
- address: 8.8.4.4
type: dns

16
manifests/site/test-site/shared/baremetalhost/node02/kustomization.yaml
View File

@ -1,16 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- baremetalhost.yaml
generatorOptions:
disableNameSuffixHash: true
secretGenerator:
- name: node02-network-data
files:
- networkData
- name: node02-bmc-secret
literals:
- username=username
- password=password

29
manifests/site/test-site/shared/baremetalhost/node02/networkData
View File

@ -1,29 +0,0 @@
links:
- id: enp0s3
name: enp0s3
type: phy
mtu: "1500"
- id: enp0s4
name: enp0s4
type: phy
mtu: "1500"
networks:
- id: private-ipv4
type: ipv4
link: enp0s3
ip_address: 10.23.25.101
netmask: 255.255.255.0
routes:
- network: 0.0.0.0
netmask: 0.0.0.0
gateway: 10.23.25.1
- id: private-ipv4
type: ipv4
link: enp0s4
ip_address: 10.23.24.101
netmask: 255.255.255.0
services:
- address: 8.8.8.8
type: dns
- address: 8.8.4.4
type: dns

27
manifests/site/test-site/shared/catalogues/hosts.yaml Normal file
View File

@ -0,0 +1,27 @@
# Site-level host catalogue. This info feeds the Templater
# kustomize plugin config in the hostgenerator-m3 function.
apiVersion: airshipit.org/v1alpha1
kind: VariableCatalogue
metadata:
name: host-catalogue
hosts:
m3:
node01:
macAddress: 52:54:00:b6:ed:31
bmcAddress: redfish+http://10.23.25.1:8000/redfish/v1/Systems/air-target-1
bmcUsername: root
bmcPassword: r00tme
ipAddresses:
oam-ipv4: 10.23.25.102
pxe-ipv4: 10.23.24.102
macAddresses:
oam: 52:54:00:9b:27:4c
pxe: 52:54:00:b6:ed:31
node02:
macAddress: 00:3b:8b:0c:ec:8b
bmcAddress: redfish+http://10.23.25.2:8000/redfish/v1/Systems/air-target-2
bmcUsername: username
bmcPassword: password
ipAddresses:
oam-ipv4: 10.23.25.101
pxe-ipv4: 10.23.24.101

5
manifests/site/test-site/shared/catalogues/kustomization.yaml Normal file
View File

@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../../type/gating/shared/catalogues
- hosts.yaml

39
manifests/type/gating/shared/catalogues/common-networking.yaml Normal file
View File

@ -0,0 +1,39 @@
# Type-level networking catalogue. This info feeds the Templater
# kustomize plugin config in the hostgenerator-m3 function.
apiVersion: airshipit.org/v1alpha1
kind: VariableCatalogue
metadata:
name: common-networking-catalogue
commonNetworking:
links:
- id: oam
name: enp0s3
type: phy
mtu: "1500"
# ethernet_mac_address: <from host-catalogue> (optional)
- id: pxe
name: enp0s4
type: phy
mtu: "1500"
# ethernet_mac_address: <from host-catalogue> (optional)
networks:
- id: oam-ipv4
type: ipv4
link: oam
# ip_address: <from host-catalogue>
netmask: 255.255.255.0
routes:
- network: 0.0.0.0
netmask: 0.0.0.0
gateway: 10.23.25.1
- id: pxe-ipv4
type: ipv4
link: pxe
# ip_address: <from host-catalogue>
netmask: 255.255.255.0
services:
- address: 8.8.8.8
type: dns
- address: 8.8.4.4
type: dns

4
manifests/type/gating/shared/catalogues/kustomization.yaml Normal file
View File

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- common-networking.yaml

17
pkg/document/plugin/templater/v1alpha1/templater.go
View File

@ -48,9 +48,24 @@ func New(_ *environment.AirshipCTLSettings, cfg []byte) (plugtypes.Plugin, error
// Run templater plugin
func (t *Templater) Run(_ io.Reader, out io.Writer) error {
tmpl, err := template.New("tmpl").Funcs(sprig.TxtFuncMap()).Parse(t.Template)
funcMap := sprig.TxtFuncMap()
funcMap["toYaml"] = toYaml
tmpl, err := template.New("tmpl").Funcs(funcMap).Parse(t.Template)
if err != nil {
return err
}
return tmpl.Execute(out, t.Values)
}
// Render input yaml as output yaml
// This function is from the Helm project:
// https://github.com/helm/helm
// Copyright The Helm Authors
func toYaml(v interface{}) string {
data, err := yaml.Marshal(v)
if err != nil {
// Swallow errors inside of a template.
return ""
}
return string(data)
}

33
pkg/document/plugin/templater/v1alpha1/templater_test.go
View File

@ -77,6 +77,39 @@ spec:
cfg: `
apiVersion: airshipit.org/v1alpha1
kind: Templater
metadata:
name: notImportantHere
values:
test:
of:
- toYaml
template: |
{{ toYaml . -}}
`,
expectedOut: `test:
of:
- toYaml
`,
},
{
cfg: `
apiVersion: airshipit.org/v1alpha1
kind: Templater
metadata:
name: notImportantHere
values:
test:
of:
- badToYamlInput
template: |
{{ toYaml ignorethisbadinput -}}
`,
expectedOut: ``,
},
{
cfg: `
apiVersion: airshipit.org/v1alpha1
kind: Templater
metadata:
name: notImportantHere
template: |

16
playbooks/airship-airshipctl-build-gate.yaml
View File

@ -36,6 +36,12 @@
include_role:
name: install-kustomize
# Installs kustomize plugin functionality needed for rendering below
- name: install airshipctl
shell: |
cd {{ local_src_dir | default(zuul.project.src_dir) }}
./tools/deployment/21_systemwide_executable.sh
- name: make sure serve directory exists
file:
dest: "{{ serve_dir }}"
@ -57,22 +63,26 @@
- name: get BareMetalHost objects
shell: |
set -e
kustomize build {{ airship_config_manifest_directory }}/{{ airship_config_site_path }}/ephemeral/controlplane |
kustomize build --enable_alpha_plugins \
{{ airship_config_manifest_directory }}/{{ airship_config_site_path }}/ephemeral/controlplane |
kustomize config grep "kind=BareMetalHost"
register: bmh_command
failed_when: "bmh_command.stdout == ''"
environment:
KUSTOMIZE_PLUGIN_HOME: "{{ ansible_env.HOME }}/.airship/kustomize-plugins"
KUSTOMIZE_ENABLE_ALPHA_COMMANDS: "true"
- set_fact:
bmh: "{{ bmh_command.stdout | from_yaml_all | list }}"
- name: get network configuration for BareMetalHost objects
shell: |
set -e
kustomize build {{ airship_config_manifest_directory }}/{{ airship_config_site_path }}/ephemeral/controlplane |
kustomize build --enable_alpha_plugins \
{{ airship_config_manifest_directory }}/{{ airship_config_site_path }}/ephemeral/controlplane |
kustomize config grep "metadata.name={{ item.spec.networkData.name }}"
register: netdata_command
failed_when: "netdata_command.stdout == ''"
environment:
KUSTOMIZE_PLUGIN_HOME: "{{ ansible_env.HOME }}/.airship/kustomize-plugins"
KUSTOMIZE_ENABLE_ALPHA_COMMANDS: "true"
with_items: "{{ bmh }}"
- name: get links from network data per BareMetalHost object
@ -81,7 +91,7 @@
{{
netdata_command.results |
map(attribute='stdout')| map('from_yaml') |
map(attribute='data.networkData') | map('b64decode') | map('from_yaml') |
map(attribute='stringData.networkData') | map('from_yaml') |
map(attribute='links') | list
}}
- name: define list of VM mac addresses

3
tools/deployment/21_systemwide_executable.sh
View File

@ -34,3 +34,6 @@ else
echo "Airshipctl version"
airshipctl version
fi
echo "Install airshipctl as kustomize plugins"
AIRSHIPCTL="/usr/local/bin/airshipctl" ./tools/document/build_kustomize_plugin.sh

4
tools/deployment/22_test_configs.sh
View File

@ -42,8 +42,8 @@ export AIRSHIP_CONFIG_EPHEMERAL_IP=${IP_Ephemeral:-"10.23.25.101"}
export AIRSHIP_CONFIG_CLIENT_CERT_DATA=$(cat tools/deployment/certificates/airship_config_client_cert_data| base64 -w0)
export AIRSHIP_CONFIG_CLIENT_KEY_DATA=$(cat tools/deployment/certificates/airship_config_client_key_data| base64 -w0)
#Remove and Create .airship folder
rm -rf $HOME/.airship
# Remove the contents of the .airship folder, preserving the kustomize plugin directory
rm -rf $HOME/.airship/*config*
mkdir -p $HOME/.airship
echo "Generate ~/.airship/config and ~/.airship/kubeconfig"

22
tools/document/build_kustomize_plugin.sh
View File

@ -32,14 +32,26 @@ fi
rm -rf ${KUSTOMIZE_PLUGIN_HOME}/airshipit.org
# copy our plugin to the PLUGIN_ROOT, and give a kustomzie-friendly wrapper
PLUGIN_PATH=${KUSTOMIZE_PLUGIN_HOME}/airshipit.org/v1alpha1/replacementtransformer
mkdir -p ${PLUGIN_PATH}
cat > ${PLUGIN_PATH}/ReplacementTransformer <<EOF
for PLUGIN in ReplacementTransformer Templater; do
PLUGIN_PATH=${KUSTOMIZE_PLUGIN_HOME}/airshipit.org/v1alpha1/$(echo ${PLUGIN} | awk '{print tolower($0)}')
mkdir -p ${PLUGIN_PATH}
cat > ${PLUGIN_PATH}/${PLUGIN} <<EOF
#!/bin/bash
\$(dirname \$0)/airshipctl document plugin "\$@"
EOF
chmod +x ${PLUGIN_PATH}/ReplacementTransformer
cp -p ${AIRSHIPCTL} ${PLUGIN_PATH}/
chmod +x ${PLUGIN_PATH}/${PLUGIN}
cp -p ${AIRSHIPCTL} ${PLUGIN_PATH}/
done
# make a fake "variablecatalogue" no-op plugin, so kustomize
# doesn't barf on leftover catalogues that were used to construct other transformer configs
PLUGIN_PATH=${KUSTOMIZE_PLUGIN_HOME}/airshipit.org/v1alpha1/variablecatalogue
mkdir -p ${PLUGIN_PATH}
cat > ${PLUGIN_PATH}/VariableCatalogue <<EOF
#!/bin/bash
# This is a no-op kustomize plugin
EOF
chmod +x ${PLUGIN_PATH}/VariableCatalogue
# tell the user how to use this
echo -e "The airshipctl kustomize plugin has been installed.\nRun kustomize with:"

21
tools/document/validate_site_docs.sh
View File

@ -114,16 +114,21 @@ for cluster in ephemeral target; do
ignore=$(for i in $phases; do echo "-I $i "; done)
phases+=$(ls $ignore manifests/site/${SITE}/${cluster}| grep -v "\.yaml$")
for phase in $phases; do
echo -e "\n*** Rendering ${cluster}/${phase}"
# Guard against bootstrap or initinfra being missing, which could be the case for some configs
if [ -d "manifests/site/${SITE}/${cluster}/${phase}" ]; then
echo -e "\n*** Rendering ${cluster}/${phase}"
# step 1: actually apply all crds in the phase
${ACTL} phase render ${phase} -k CustomResourceDefinition > ${TMP}/crds.yaml
if [ -s ${TMP}/crds.yaml ]; then
${KUBECTL} --context ${CONTEXT} --kubeconfig ${KUBECONFIG} apply -f ${TMP}/crds.yaml
# step 1: actually apply all crds in the phase
# TODO: will need to loop through phases in order, eventually
# e.g., load CRDs from initinfra first, so they're present when validating later phases
${ACTL} phase render ${phase} -k CustomResourceDefinition > ${TMP}/crds.yaml
if [ -s ${TMP}/crds.yaml ]; then
${KUBECTL} --context ${CONTEXT} --kubeconfig ${KUBECONFIG} apply -f ${TMP}/crds.yaml
fi
# step 2: dry-run the entire phase
${ACTL} phase apply --dry-run ${phase}
fi
# step 2: dry-run the entire phase
${ACTL} phase apply --dry-run ${phase}
done
fi
done

2
zuul.d/jobs.yaml
View File

@ -90,7 +90,7 @@
site_name: test-site
gate_scripts:
- ./tools/deployment/01_install_kubectl.sh
- ./tools/deployment/21_systemwide_executable.sh
# 21_systemwide_executable.sh is run in the build-gate pre-run above
- ./tools/deployment/22_test_configs.sh
- ./tools/deployment/24_build_ephemeral_iso.sh
- ./tools/deployment/25_deploy_ephemeral_node.sh