Add a Hiera element

Provides a way to configure hiera.yaml and deploy custom
JSON or YAML hiera datafiles via Heat. There are 3
ways to get data into Hiera files:

  -static data
  -mapped data
  -OAC (os-apply-config) mapped data

By default the mapped and OAC data formats use YAML.
Static data files can be in either format.

Change-Id: I3a3eb8382b34112784799f48bcf86015ea56f304

elements/hiera/README.md

@@ -0,0 +1,65 @@
+Configure Hiera for use w/ Puppet and Heat metadata.
+
+Configuration
+-------------
+
+    hiera:
+      hierarchy: []
+        - The order to load datafiles. This is configured in hiera.yaml.
+
+      datafiles: {}
+          Data files is a hash of filename -> {data/mappings} to inject
+          into each named hiera datafile. There are three types:
+
+        raw_data: contains static raw data to inject directly into this hiera
+          datafile. Can be an inline string or imported via get_file in a
+          Heat template.
+
+        mapped_data: Name value pairs that will be injected into the
+          hiera data file. Hiera name on the left, Hiera value on the right.
+
+        oac_data: A hash of puppet -> OAC (os-apply-config) named key value
+          pairs. Arbitrary os-apply-config data can be mapped to Hiera keys
+          using this data. Hiera name on the left. The Hiera value is generated
+          at os-refresh-config time via os-apply-config lookup of the value.
+
+Example:
+--------
+
+    HieraConfig:
+      type: OS::Heat::StructuredConfig
+      properties:
+        group: os-apply-config
+        config:
+          hiera:
+            hierarchy:
+              - heat_config_%{::deploy_config_name}
+              - controller
+              - common
+            datafiles:
+              controller:
+                raw_data: {get_file: puppet/hieradata/controller.yaml}
+                oac_data:
+                  bootstrap_nodeid: bootstrap_host.bootstrap_nodeid
+              common:
+                data: {get_file: puppet/hieradata/common.yaml}
+
+    HieraDeployment:
+      type: OS::Heat::StructuredDeployment
+      properties:
+        server: {get_resource: MyServer}
+        config: {get_resource: HieraConfig}
+        signal_transport: NO_SIGNAL
+
+In this example the 'hierarchy' config section controls the ordering of the
+hiera files within hiera.yaml. The 'datafiles' config section controls the
+actual hiera data files which gets injected into the node.
+
+NOTE:
+This example makes use of a special heat\_config\_%{::deploy\_config\_name}
+heira datafile which gets generated via the heat-config-puppet element when
+enable\_hiera is set to True. Since this file is injected automatically
+we don't specify it in 'datafiles' but we do have it listed in the 'hierarchy'.
+A FACTER\_ variable is used to provide access to the ::deploy\_config\_name
+variable which is automatically set via the heat-config-puppet element when
+puppet apply is executed.

elements/hiera/element-deps

@@ -0,0 +1,4 @@
+os-apply-config
+os-refresh-config
+package-installs
+puppet

elements/hiera/install.d/package-installs-hiera

@@ -0,0 +1 @@
+jq

elements/hiera/os-apply-config/etc/puppet/hiera.yaml

@@ -0,0 +1,12 @@
+---
+:backends:
+  - json
+  - yaml
+:json:
+  :datadir: /etc/puppet/hieradata
+:yaml:
+  :datadir: /etc/puppet/hieradata
+:hierarchy:
+  {{#hiera.hierarchy}}
+  - {{.}}
+  {{/hiera.hierarchy}}

elements/hiera/os-refresh-config/configure.d/40-hiera-datafiles

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# Configure hiera datafiles based on Heat metadata.
+set -eu
+
+mkdir -p /etc/puppet/hieradata
+
+function write_oac_data() {
+  local key=$1
+  local filename=$2
+  # Lookup data for the associated mapping (hiera.datafiles.<name>.oac_data)
+  local HIERA_DATA=$(os-apply-config --key $key --type raw --key-default '')
+  local HIERA_DATAMAP_KEYS=$(jq keys <<< $HIERA_DATA)
+  local COUNT=$(($(jq length <<< $HIERA_DATAMAP_KEYS) - 1))
+
+  for i in $(seq 0 $COUNT); do
+    local KEY=$(jq -r ".[$i]" <<< $HIERA_DATAMAP_KEYS)
+    local OAC_KEY=$(jq -r -a ".[\"$KEY\"]" <<< $HIERA_DATA)
+    local OAC_VALUE=$(os-apply-config --key $OAC_KEY --type raw --key-default '')
+    # Quote multi-line strings for YAML
+    if [ $(echo -ne "$OAC_VALUE" | grep -c '$') -gt 1 ]; then
+      echo "$KEY: '$OAC_VALUE'" >> $filename
+    else
+      echo "$KEY: $OAC_VALUE" >> $filename
+    fi
+
+  done
+}
+
+function write_mapped_data() {
+  local key=$1
+  local filename=$2
+  # Lookup data for the associated mapping (hiera.datafiles.<name>.mapped_data)
+  local HIERA_DATA=$(os-apply-config --key $key --type raw --key-default '')
+  local HIERA_DATAMAP_KEYS=$(jq keys <<< $HIERA_DATA)
+  local COUNT=$(($(jq length <<< $HIERA_DATAMAP_KEYS) - 1))
+
+  for i in $(seq 0 $COUNT); do
+    local KEY=$(jq -r ".[$i]" <<< $HIERA_DATAMAP_KEYS)
+    local VALUE=$(jq -r -a ".[\"$KEY\"]" <<< $HIERA_DATA)
+    # Quote multi-line strings for YAML
+    if [ $(echo -ne "$VALUE" | grep -c '$') -gt 1 ]; then
+      echo "$KEY: '$VALUE'" >> $filename
+    else
+      echo "$KEY: $VALUE" >> $filename
+    fi
+  done
+}
+
+# Loop over all the datafiles
+HIERA_DATAFILES=$(os-apply-config --key hiera.datafiles --type raw --key-default '')
+HIERA_DATAFILE_KEYS=$(jq keys <<< $HIERA_DATAFILES)
+COUNT=$(($(jq length <<< $HIERA_DATAFILE_KEYS) - 1))
+for i in $(seq 0 $COUNT); do
+    KEY=$(jq -r ".[$i]" <<< $HIERA_DATAFILE_KEYS)
+    FILENAME="/etc/puppet/hieradata/$KEY.yaml"
+    # First we write out any static data
+    HIERA_DATA=$(os-apply-config --key hiera.datafiles.$KEY.raw_data --type raw --key-default '')
+    echo -e "$HIERA_DATA" > /etc/puppet/hieradata/$KEY.yaml
+    write_mapped_data "hiera.datafiles.${KEY}.mapped_data" $FILENAME
+    write_oac_data "hiera.datafiles.${KEY}.oac_data" $FILENAME
+done