functional-test : ensure glean started

We use glean in the gate tests to enable the network.  Because we are
in a DHCP environment with the devstack environment, it's possible
that glean didn't start but we still manage to get an address anyway
via autoconfiguration.  But the point of having glean is that in
clouds without DHCP things work too.  This is most common with new
distributions, where glean matching may not have updated to configure
things fully.

We could replicate a non-DHCP environment, at some considerable
initial cost of figuring out how to setup devstack to do that and then
ongoing costs of having a fairly bespoke environment.  A simpler path
is to just check the glean services and make sure they started
correctly.

Change-Id: Idb8d67cd2ba6e8c5d7157177e9cfd6be7b99cacd

tools/functional-test-check.sh

@@ -86,6 +86,22 @@ function sshintonode {
        FAILURE_REASON="Failed to find correct kernel boot flags $node"
        RETURN=1
     fi
+
+    # Ensure glean services have loaded
+    # The output is like:
+    # ---
+    #  glean-early.service loaded active exited    Early glean execution
+    #  glean@ens3.service  loaded active exited    Glean for interface ens3 with NetworkManager
+    #  glean@lo.service    loaded active exited    Glean for interface lo with NetworkManager
+    # ---
+    # So if we see anything other than 'loaded active exited' we have a problem.
+    glean_status=$(/tmp/ssh_wrapper $node -- "systemctl | egrep 'glean[@|-]' | { grep -v 'loaded active exited' || true; }")
+    if [[ ${glean_status} != '' ]]; then
+        echo "*** Glean not loaded correctly"
+        echo "*** saw: ${glean_status}"
+        FAILURE_REASON="Failed to start glean correctly"
+        RETURN=1
+    fi
 }
 
 function checknm {