Using cgroupfs as default cgroup-driver

This is a part of fixes for k8s v1.11.1 recently we're doing. When
testing the k8s v1.11.1, we just found some small but annoying issues:

1. cgroup-driver with systemd not working well with Fedora Atomic, so
   we're going to use cgroupfs as the default cgroup-driver.
2. The $ char need to be escaped wc-notify-master.sh

Task: 23223
Story: 2003103

Change-Id: I995f5b82abadfdb7f78f7c098ac7a7f1e5c34fd3

doc/source/user/index.rst

@@ -356,8 +356,9 @@ the table are linked to more details elsewhere in the user guide.
 | `kubeproxy_options`_                  | extra kubeproxy    | ""            |
 |                                       | args               |               |
 +---------------------------------------+--------------------+---------------+
-| `cgroup_driver`_                      | - systemd          | "systemd"     |
+| `cgroup_driver`_                      | - systemd          | "cgroupfs"    |
 |                                       | - cgroupfs         |               |
++---------------------------------------+--------------------+---------------+
 | `cloud_provider_enabled`_             | - true             | true          |
 |                                       | - false            |               |
 +---------------------------------------+--------------------+---------------+
@@ -2167,6 +2168,10 @@ Calico needs extra mounts for the kubelet container. See `commit
 <https://github.com/projectatomic/atomic-system-containers/commit/54ab8abc7fa1bfb6fa674f55cd0c2fa0c812fd36>`_
 of atomic-system-containers for more information.
 
+**NOTE:** We have seen some issues using systemd as cgroup-driver with Calico
+together, so we highly recommend to use cgroupfs as the cgroup-driver for
+Calico.
+
 Network for VMs
 ---------------
 

magnum/drivers/common/templates/kubernetes/fragments/wc-notify-master.sh

@@ -13,7 +13,7 @@ WC_NOTIFY_SERVICE=/etc/systemd/system/wc-notify.service
 
 cat > $WC_NOTIFY_BIN <<EOF
 #!/bin/bash -v
-until  [ "ok" = "$(curl --silent http://127.0.0.1:8080/healthz)" ]
+until  [ "ok" = "\$(curl --silent http://127.0.0.1:8080/healthz)" ]
 do
     echo "Waiting for Kubernetes API..."
     sleep 5

magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml

@@ -159,7 +159,7 @@ parameters:
     description: >
       cgroup driver name that kubelet should use, ideally the same as
       the docker cgroup driver.
-    default: "systemd"
+    default: "cgroupfs"
 
   wait_condition_timeout:
     type: number

magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml

@@ -66,7 +66,7 @@ parameters:
     description: >
       cgroup driver name that kubelet should use, ideally the same as
       the docker cgroup driver.
-    default: "systemd"
+    default: "cgroupfs"
 
   volume_driver:
     type: string

magnum/drivers/k8s_fedora_atomic_v1/templates/kubeminion.yaml

@@ -56,7 +56,7 @@ parameters:
     description: >
       cgroup driver name that kubelet should use, ideally the same as
       the docker cgroup driver.
-    default: "systemd"
+    default: "cgroupfs"
 
   tls_disabled:
     type: boolean