Ensure redis_tls_proxy starts after all redis instances

When converting a HA control plane to TLS-e, 1) the bootstrap node
tells pacemaker to restart all redis instances to take into
account the new TLS-e config; 2) a new container redis_tls_proxy
is started on every controller to encapsulate redis traffic in TLS
tunnels. This happens during step 2.

Redis servers have to be restarted everywhere for redis_tls_proxy
to be able to start tunnels properly. Since we can't guarantee that
across several nodes during the same step, tweak the startup of
redis_tls_proxy instead; make sure to only create the tunnels once
the targeted host:port can be bound (i.e. redis was restarted).

Change-Id: I70560f80775dacddd82262e8079c13f86b0eb0e6
Closes-Bug: #1883096

container_config_scripts/wait-port-and-run.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+set -eu
+
+HOST=$1
+PORT=$2
+
+echo "$(date -u): Checking whether we can bind to ${HOST}:${PORT}"
+while (ss -Htnl src "${HOST}" "sport = :${PORT}" | grep -wq "${PORT}"); do
+    echo "$(date -u): ${HOST}:${PORT} still in use, waiting...";
+    sleep 10;
+done
+
+shift 2
+COMMAND="$*"
+if [ -z "${COMMAND}" ]; then
+    COMMAND="true"
+fi
+exec $COMMAND

deployment/containers-common.yaml

@@ -118,6 +118,9 @@ outputs:
             template: { get_file: ../container_config_scripts/pacemaker_wait_bundle.sh }
             params:
               __PCMKTIMEOUT__: {get_param: PcmkConfigRestartTimeout}
+      wait-port-and-run.sh:
+        mode: "0755"
+        content: { get_file: ../container_config_scripts/wait-port-and-run.sh }
 
   volumes_base:
     description: Base volume list

deployment/database/redis-pacemaker-puppet.yaml

@@ -206,7 +206,14 @@ outputs:
               owner: redis:redis
               recurse: true
         /var/lib/kolla/config_files/redis_tls_proxy.json:
-          command: stunnel /etc/stunnel/stunnel.conf
+          command:
+            # Note: kolla doesn't process string arguments as expected,
+            # so use a bash idiom to achieve the same result
+            str_replace:
+              template:
+                bash -c $* -- eval /wait-port-and-run.sh $(hiera fqdn_$NETWORK) 6379 stunnel /etc/stunnel/stunnel.conf
+              params:
+                $NETWORK: {get_param: [ServiceNetMap, RedisNetwork]}
           config_files:
             - source: "/var/lib/kolla/config_files/src/*"
               dest: "/"
@@ -290,6 +297,7 @@ outputs:
                           - /var/lib/config-data/puppet-generated/redis:/var/lib/kolla/config_files/src:ro
                           - /etc/pki/tls/certs/redis.crt:/var/lib/kolla/config_files/src-tls/etc/pki/tls/certs/redis.crt:ro
                           - /etc/pki/tls/private/redis.key:/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/redis.key:ro
+                          - /var/lib/container-config-scripts/wait-port-and-run.sh:/wait-port-and-run.sh:ro
                     environment:
                       KOLLA_CONFIG_STRATEGY: COPY_ALWAYS
                 - {}

tools/yaml-validate.py

@@ -1029,6 +1029,10 @@ def validate_service_hiera_interpol(f, tpl):
             if enter_lists and path[-1] != 0 and path[-2] != 'get_param':
                 continue
 
+            # Omit if it is not a hiera config setting
+            if path[1] in ['kolla_config']:
+                continue
+
             path_str = ';'.join(str(x) for x in path)
             # NOTE(bogdando): Omit foo_network keys looking like a network
             # name. The only exception is allow anything under