Run etcd and flanneld in a system container

In Fedora Atomic 27 etcd and flanneld are removed from the base image.
Install them as a system containers.

* update docker-storage configuration
* add etcd and flannel tags as labels

Change-Id: I2103c7c3d50f4b68ddc11abff72bc9e3f22839f3
Closes-Bug: #1735381

devstack/plugin.sh

@@ -13,7 +13,7 @@ if is_service_enabled magnum-api magnum-cond; then
         echo_summary "Installing magnum"
         install_magnum
 
-        MAGNUM_GUEST_IMAGE_URL=${MAGNUM_GUEST_IMAGE_URL:-"https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2"}
+        MAGNUM_GUEST_IMAGE_URL=${MAGNUM_GUEST_IMAGE_URL:-"https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2"}
         IMAGE_URLS+=",${MAGNUM_GUEST_IMAGE_URL}"
 
         LIBS_FROM_GIT="${LIBS_FROM_GIT},python-magnumclient"

doc/source/contributor/quickstart.rst

@@ -208,7 +208,7 @@ when installing devstack use::
     +--------------------------------------+------------------------------------+--------+
     | ID                                   | Name                               | Status |
     +--------------------------------------+------------------------------------+--------+
-    | 0bc132b1-ee91-4bd8-b0fd-19deb57fb39f | Fedora-Atomic-26-20170723.0.x86_64 | active |
+    | 0bc132b1-ee91-4bd8-b0fd-19deb57fb39f | Fedora-Atomic-27-20180212.2.x86_64 | active |
     | 7537bbf2-f1c3-47da-97bb-38c09007e146 | cirros-0.3.5-x86_64-disk           | active |
     +--------------------------------------+------------------------------------+--------+
 
@@ -253,7 +253,7 @@ Atomic image so the clusters which use this ClusterTemplate will be based on
 Fedora Atomic::
 
     openstack coe cluster template create k8s-cluster-template \
-                           --image Fedora-Atomic-26-20170723.0.x86_64 \
+                           --image Fedora-Atomic-27-20180212.2.x86_64 \
                            --keypair testkey \
                            --external-network public \
                            --dns-nameserver 8.8.8.8 \
@@ -529,7 +529,7 @@ except for the absence of some Kubernetes-specific arguments and the use of
 'swarm' as the COE::
 
     openstack coe cluster template create swarm-cluster-template \
-                           --image Fedora-Atomic-26-20170723.0.x86_64 \
+                           --image Fedora-Atomic-27-20180212.2.x86_64 \
                            --keypair testkey \
                            --external-network public \
                            --dns-nameserver 8.8.8.8 \

doc/source/install/launch-instance.rst

@@ -116,7 +116,7 @@ and tested by the Magnum team.
 
    .. code-block:: console
 
-      $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2
+      $ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2
 
 #. Register the image to the Image service setting the ``os_distro`` property
    to ``fedora-atomic``:
@@ -126,7 +126,7 @@ and tested by the Magnum team.
       $ openstack image create \
                             --disk-format=qcow2 \
                             --container-format=bare \
-                            --file=Fedora-Atomic-26-20170723.0.x86_64.qcow2\
+                            --file=Fedora-Atomic-27-20180212.2.x86_64.qcow2\
                             --property os_distro='fedora-atomic' \
                             fedora-atomic-latest
       +------------------+------------------------------------------------------+

doc/source/user/index.rst

@@ -338,6 +338,10 @@ the table are linked to more details elsewhere in the user guide.
 +---------------------------------------+--------------------+---------------+
 | `kube_tag`_                           | see below          | see below     |
 +---------------------------------------+--------------------+---------------+
+| `etcd_tag`_                           | see below          | see below     |
++---------------------------------------+--------------------+---------------+
+| `flannel_tag`_                        | see below          | see below     |
++---------------------------------------+--------------------+---------------+
 | `kube_dashboard_enabled`_             | - true             | true          |
 |                                       | - false            |               |
 +---------------------------------------+--------------------+---------------+
@@ -1098,6 +1102,8 @@ _`container_infra_prefix`
   * docker.io/openstackmagnum/kubernetes-kubelet
   * docker.io/openstackmagnum/kubernetes-proxy
   * docker.io/openstackmagnum/kubernetes-scheduler
+  * docker.io/openstackmagnum/etcd
+  * docker.io/openstackmagnum/flannel
   * docker.io/prom/node-exporter:latest
   * docker.io/prom/prometheus:latest
   * gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
@@ -1110,6 +1116,20 @@ _`kube_tag`
   If unset, the current Magnum version's default Kubernetes release is
   installed.
 
+_`etcd_tag`
+  This label allows users to select `a specific etcd version,
+  based on its container tag
+  <https://hub.docker.com/r/openstackmagnum/etcd/tags/>`_.
+  If unset, the current Magnum version's a default etcd version.
+  For queens, v3.2.7
+
+_`flannel_tag`
+  This label allows users to select `a specific flannel version,
+  based on its container tag
+  <https://hub.docker.com/r/openstackmagnum/flannel/tags/>`_.
+  If unset, the current Magnum version's a default flannel version.
+  For queens, v0.9.0
+
 _`kube_dashboard_enabled`
   This label triggers the deployment of the kubernetes dashboard.
   The default value is 1, meaning it will be enabled.

magnum/drivers/common/templates/fragments/configure_docker_storage_driver_atomic.sh

@@ -7,10 +7,9 @@
 clear_docker_storage () {
     # stop docker
     systemctl stop docker
+    systemctl disable docker-storage-setup
     # clear storage graph
     rm -rf /var/lib/docker/*
-    # remove current LVs
-    docker-storage-setup --reset
 
     if [ -f /etc/sysconfig/docker-storage ]; then
         sed -i "/^DOCKER_STORAGE_OPTIONS=/ s/=.*/=/" /etc/sysconfig/docker-storage
@@ -27,11 +26,7 @@ configure_storage_driver_generic() {
         mount -a
     fi
 
-    sed -i "/^DOCKER_STORAGE_OPTIONS=/ s/=.*/=-s $1/" /etc/sysconfig/docker-storage
+    echo "DOCKER_STORAGE_OPTIONS=\"--storage-driver $1\"" > /etc/sysconfig/docker-storage
-
-    local lvname=$(lvdisplay | grep "LV\ Path" | awk '{print $3}')
-    local pvname=$(pvdisplay | grep "PV\ Name" | awk '{print $3}')
-    lvextend -r $lvname $pvname
 }
 
 # Configure docker storage with devicemapper using direct LVM
@@ -39,7 +34,7 @@ configure_devicemapper () {
     clear_docker_storage
 
     echo "GROWROOT=True" > /etc/sysconfig/docker-storage-setup
-    echo "ROOT_SIZE=5GB" >> /etc/sysconfig/docker-storage-setup
+    echo "STORAGE_DRIVER=devicemapper" >> /etc/sysconfig/docker-storage-setup
 
     if [ -n "$DOCKER_VOLUME_SIZE" ] && [ "$DOCKER_VOLUME_SIZE" -gt 0 ]; then
 
@@ -48,6 +43,7 @@ configure_devicemapper () {
 
         echo "VG=docker" >> /etc/sysconfig/docker-storage-setup
     else
+        echo "ROOT_SIZE=5GB" >> /etc/sysconfig/docker-storage-setup
         echo "DATA_SIZE=95%FREE" >> /etc/sysconfig/docker-storage-setup
     fi
 

magnum/drivers/common/templates/kubernetes/fragments/configure-etcd.sh

@@ -34,13 +34,20 @@ if [ -n "$ETCD_VOLUME_SIZE" ] && [ "$ETCD_VOLUME_SIZE" -gt 0 ]; then
 
 fi
 
+_prefix=${CONTAINER_INFRA_PREFIX:-docker.io/openstackmagnum/}
+atomic install \
+--system-package no \
+--system \
+--storage ostree \
+--name=etcd ${_prefix}etcd:${ETCD_TAG}
+
 if [ -z "$KUBE_NODE_IP" ]; then
     # FIXME(yuanying): Set KUBE_NODE_IP correctly
     KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
 fi
 
 myip="${KUBE_NODE_IP}"
-cert_dir="/etc/kubernetes/certs"
+cert_dir="/etc/etcd/certs"
 protocol="https"
 
 if [ "$TLS_DISABLED" = "True" ]; then

magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh

@@ -17,30 +17,16 @@ atomic install --storage ostree --system --system-package=no --name=kube-proxy $
 
 CERT_DIR=/etc/kubernetes/certs
 PROTOCOL=https
-FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
--etcd-certfile $CERT_DIR/proxy.crt \
--etcd-keyfile $CERT_DIR/proxy.key"
-ETCD_CURL_OPTIONS="--cacert $CERT_DIR/ca.crt \
---cert $CERT_DIR/proxy.crt --key $CERT_DIR/proxy.key"
 ETCD_SERVER_IP=${ETCD_SERVER_IP:-$KUBE_MASTER_IP}
 KUBE_PROTOCOL="https"
 KUBELET_KUBECONFIG=/etc/kubernetes/kubelet-config.yaml
 PROXY_KUBECONFIG=/etc/kubernetes/proxy-config.yaml
-FLANNELD_CONFIG=/etc/sysconfig/flanneld
 
 if [ "$TLS_DISABLED" = "True" ]; then
     PROTOCOL=http
-    FLANNEL_OPTIONS=""
-    ETCD_CURL_OPTIONS=""
     KUBE_PROTOCOL="http"
 fi
 
-sed -i '/FLANNEL_OPTIONS/'d $FLANNELD_CONFIG
-
-cat >> $FLANNELD_CONFIG <<EOF
-FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
-EOF
-
 KUBE_MASTER_URI="$KUBE_PROTOCOL://$KUBE_MASTER_IP:$KUBE_API_PORT"
 
 HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
@@ -162,9 +148,25 @@ sed -i '
 ' /etc/kubernetes/proxy
 
 if [ "$NETWORK_DRIVER" = "flannel" ]; then
-    sed -i '
+    atomic install --storage ostree --system --system-package=no \
-        /^FLANNEL_ETCD_ENDPOINTS=/ s|=.*|="'"$PROTOCOL"'://'"$ETCD_SERVER_IP"':2379"|
+    --name=flanneld ${_prefix}flannel:${FLANNEL_TAG}
-    ' $FLANNELD_CONFIG
+    if [ "$TLS_DISABLED" = "True" ]; then
+        FLANNEL_OPTIONS=""
+        ETCD_CURL_OPTIONS=""
+    else
+        FLANNEL_CERT_DIR=/etc/flanneld/certs
+        FLANNEL_OPTIONS="-etcd-cafile $FLANNEL_CERT_DIR/ca.crt"
+        FLANNEL_OPTIONS="$FLANNEL_OPTIONS -etcd-certfile $FLANNEL_CERT_DIR/proxy.crt"
+        FLANNEL_OPTIONS="$FLANNEL_OPTIONS -etcd-keyfile $FLANNEL_CERT_DIR/proxy.key"
+        ETCD_CURL_OPTIONS="--cacert $FLANNEL_CERT_DIR/ca.crt --cert $FLANNEL_CERT_DIR/proxy.crt --key $FLANNEL_CERT_DIR/proxy.key"
+    fi
+    FLANNELD_CONFIG=/etc/sysconfig/flanneld
+
+    cat >> $FLANNELD_CONFIG <<EOF
+    FLANNEL_ETCD_ENDPOINTS="$PROTOCOL://${ETCD_SERVER_IP}:2379"
+    FLANNEL_ETCD_PREFIX="/atomic.io/network"
+    FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
+EOF
 
     # Make sure etcd has a flannel configuration
     . $FLANNELD_CONFIG

magnum/drivers/common/templates/kubernetes/fragments/make-cert-client.sh

@@ -141,3 +141,5 @@ chmod 550 "${cert_dir}"
 chown -R kube:kube_etcd "${cert_dir}"
 chmod 440 ${cert_dir}/kubelet.key
 chmod 440 ${cert_dir}/proxy.key
+mkdir -p /etc/flanneld/certs
+cp ${cert_dir}/* /etc/flanneld/certs

magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh

@@ -141,3 +141,5 @@ usermod -a -G kube_etcd kube
 chmod 550 "${cert_dir}"
 chown -R kube:kube_etcd "${cert_dir}"
 chmod 440 $SERVER_KEY
+mkdir -p /etc/etcd/certs
+cp ${cert_dir}/* /etc/etcd/certs

magnum/drivers/common/templates/kubernetes/fragments/network-config-service.sh

@@ -7,25 +7,15 @@ if [ "$NETWORK_DRIVER" != "flannel" ]; then
 fi
 CERT_DIR=/etc/kubernetes/certs
 PROTOCOL=https
-FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
--etcd-certfile $CERT_DIR/server.crt \
--etcd-keyfile $CERT_DIR/server.key"
 ETCD_CURL_OPTIONS="--cacert $CERT_DIR/ca.crt \
 --cert $CERT_DIR/server.crt --key $CERT_DIR/server.key"
 FLANNELD_CONFIG=/etc/sysconfig/flanneld
 
 if [ "$TLS_DISABLED" = "True" ]; then
     PROTOCOL=http
-    FLANNEL_OPTIONS=""
     ETCD_CURL_OPTIONS=""
 fi
 
-sed -i '/FLANNEL_OPTIONS/'d $FLANNELD_CONFIG
-
-cat >> $FLANNELD_CONFIG <<EOF
-FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
-EOF
-
 . $FLANNELD_CONFIG
 
 FLANNEL_CONFIG_BIN=/usr/local/bin/flannel-config

magnum/drivers/common/templates/kubernetes/fragments/network-service.sh

@@ -43,7 +43,7 @@ Requires=flanneld.service
 
 [Service]
 Type=oneshot
-EnvironmentFile=/run/flannel/subnet.env
+EnvironmentFile=/run/flanneld/subnet.env
 ExecStart=/usr/local/bin/flannel-docker-bridge
 
 [Install]
@@ -79,4 +79,4 @@ chmod 0644 $FLANNEL_DOCKER_BRIDGE_CONF
 
 echo "activating service flanneld"
 systemctl enable flanneld
-systemctl --no-block start flanneld
+systemctl start flanneld

magnum/drivers/common/templates/kubernetes/fragments/write-heat-params-master.yaml

@@ -40,6 +40,7 @@ write_files:
       NO_PROXY="$NO_PROXY"
       WAIT_CURL="$WAIT_CURL"
       KUBE_TAG="$KUBE_TAG"
+      ETCD_TAG="$ETCD_TAG"
       KUBE_VERSION="$KUBE_VERSION"
       KUBE_DASHBOARD_VERSION="$KUBE_DASHBOARD_VERSION"
       TRUSTEE_USER_ID="$TRUSTEE_USER_ID"

magnum/drivers/common/templates/kubernetes/fragments/write-heat-params.yaml

@@ -37,6 +37,7 @@ write_files:
       NO_PROXY="$NO_PROXY"
       WAIT_CURL="$WAIT_CURL"
       KUBE_TAG="$KUBE_TAG"
+      FLANNEL_TAG="$FLANNEL_TAG"
       KUBE_VERSION="$KUBE_VERSION"
       TRUSTEE_USER_ID="$TRUSTEE_USER_ID"
       TRUSTEE_PASSWORD="$TRUSTEE_PASSWORD"

magnum/drivers/common/templates/kubernetes/fragments/write-kube-os-config.sh

@@ -2,6 +2,7 @@
 
 . /etc/sysconfig/heat-params
 
+mkdir -p /etc/kubernetes/
 KUBE_OS_CLOUD_CONFIG=/etc/kubernetes/kube_openstack_config
 cp /etc/pki/tls/certs/ca-bundle.crt /etc/kubernetes/ca-bundle.crt
 

magnum/drivers/common/templates/kubernetes/fragments/write-network-config.sh

@@ -6,14 +6,14 @@ if [ "$NETWORK_DRIVER" != "flannel" ]; then
     exit 0
 fi
 
-. /etc/sysconfig/flanneld
-
 FLANNEL_JSON=/etc/sysconfig/flannel-network.json
 FLANNELD_CONFIG=/etc/sysconfig/flanneld
 
-sed -i '
+cat > /etc/sysconfig/flanneld <<EOF
-    /^FLANNEL_ETCD_ENDPOINTS=/ s/=.*/="http:\/\/127.0.0.1:2379"/
+FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379"
-' /etc/sysconfig/flanneld
+FLANNEL_ETCD_PREFIX="/atomic.io/network"
+FLANNEL_OPTIONS=
+EOF
 
 # Generate a flannel configuration that we will
 # store into etcd using curl.

magnum/drivers/heat/k8s_fedora_template_def.py

@@ -94,7 +94,8 @@ class K8sFedoraTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
         label_list = ['kube_tag', 'container_infra_prefix',
                       'availability_zone',
                       'calico_tag', 'calico_cni_tag',
-                      'calico_kube_controllers_tag', 'calico_ipv4pool']
+                      'calico_kube_controllers_tag', 'calico_ipv4pool',
+                      'etcd_tag', 'flannel_tag']
         for label in label_list:
             label_value = cluster.labels.get(label)
             if label_value:

magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml

@@ -315,6 +315,16 @@ parameters:
     description: tag of the k8s containers used to provision the kubernetes cluster
     default: v1.9.3
 
+  etcd_tag:
+    type: string
+    description: tag of the etcd system container
+    default: v3.2.7
+
+  flannel_tag:
+    type: string
+    description: tag of the flannel system containers
+    default: v0.9.0
+
   kube_version:
     type: string
     description: version of kubernetes used for kubernetes cluster
@@ -586,6 +596,7 @@ resources:
           no_proxy: {get_param: no_proxy}
           kube_tag: {get_param: kube_tag}
           kube_version: {get_param: kube_version}
+          etcd_tag: {get_param: etcd_tag}
           kube_dashboard_version: {get_param: kube_dashboard_version}
           trustee_user_id: {get_param: trustee_user_id}
           trustee_password: {get_param: trustee_password}
@@ -665,6 +676,7 @@ resources:
           no_proxy: {get_param: no_proxy}
           kube_tag: {get_param: kube_tag}
           kube_version: {get_param: kube_version}
+          flannel_tag: {get_param: flannel_tag}
           trustee_user_id: {get_param: trustee_user_id}
           trustee_username: {get_param: trustee_username}
           trustee_password: {get_param: trustee_password}

magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml

@@ -211,6 +211,10 @@ parameters:
     type: string
     description: tag of the k8s containers used to provision the kubernetes cluster
 
+  etcd_tag:
+    type: string
+    description: tag of the etcd system container
+
   kube_version:
     type: string
     description: version of kubernetes used for kubernetes cluster
@@ -379,6 +383,7 @@ resources:
             "$HTTPS_PROXY": {get_param: https_proxy}
             "$NO_PROXY": {get_param: no_proxy}
             "$KUBE_TAG": {get_param: kube_tag}
+            "$ETCD_TAG": {get_param: etcd_tag}
             "$KUBE_VERSION": {get_param: kube_version}
             "$KUBE_DASHBOARD_VERSION": {get_param: kube_dashboard_version}
             "$WAIT_CURL": {get_attr: [master_wait_handle, curl_cli]}
@@ -459,12 +464,6 @@ resources:
       group: ungrouped
       config: {get_file: ../../common/templates/kubernetes/fragments/enable-services-master.sh}
 
-  network_service:
-    type: OS::Heat::SoftwareConfig
-    properties:
-      group: ungrouped
-      config: {get_file: ../../common/templates/kubernetes/fragments/network-service.sh}
-
   kube_apiserver_to_kubelet_role:
     type: OS::Heat::SoftwareConfig
     properties:
@@ -516,15 +515,14 @@ resources:
         - config: {get_resource: write_heat_params}
         - config: {get_resource: configure_etcd}
         - config: {get_resource: write_kube_os_config}
-        - config: {get_resource: make_cert}
         - config: {get_resource: configure_docker_storage}
         - config: {get_resource: configure_kubernetes}
+        - config: {get_resource: make_cert}
         - config: {get_resource: add_proxy}
         - config: {get_resource: start_container_agent}
         - config: {get_resource: enable_services}
         - config: {get_resource: write_network_config}
         - config: {get_resource: network_config_service}
-        - config: {get_resource: network_service}
         - config: {get_resource: kube_apiserver_to_kubelet_role}
         - config: {get_resource: core_dns_service}
         - config: {get_resource: kube_ui_service}

magnum/drivers/k8s_fedora_atomic_v1/templates/kubeminion.yaml

@@ -175,6 +175,10 @@ parameters:
     type: string
     description: tag of the k8s containers used to provision the kubernetes cluster
 
+  flannel_tag:
+    type: string
+    description: tag of the flannel system containers
+
   kube_version:
     type: string
     description: version of kubernetes used for kubernetes cluster
@@ -300,6 +304,7 @@ resources:
             $HTTPS_PROXY: {get_param: https_proxy}
             $NO_PROXY: {get_param: no_proxy}
             $KUBE_TAG: {get_param: kube_tag}
+            $FLANNEL_TAG: {get_param: flannel_tag}
             $KUBE_VERSION: {get_param: kube_version}
             $WAIT_CURL: {get_attr: [minion_wait_handle, curl_cli]}
             $TRUSTEE_USER_ID: {get_param: trustee_user_id}

magnum/tests/contrib/gate_hook.sh

@@ -85,8 +85,8 @@ elif [ "${coe}${special}" = "k8s-ironic" ]; then
     export DEVSTACK_LOCAL_CONFIG+=$'\n'"IRONIC_VM_SPECS_DISK=10"
     export DEVSTACK_LOCAL_CONFIG+=$'\n'"IRONIC_VM_EPHEMERAL_DISK=5"
 else
-    export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_GUEST_IMAGE_URL='${NODEPOOL_ATOMIC_MIRROR}/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2'"
+    export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_GUEST_IMAGE_URL='${NODEPOOL_ATOMIC_MIRROR}/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2'"
-    export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_IMAGE_NAME='Fedora-Atomic-26-20170723.0.x86_64'"
+    export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_IMAGE_NAME='Fedora-Atomic-27-20180212.2.x86_64'"
 fi
 
 # Enable magnum plugin in the last step

magnum/tests/unit/drivers/test_template_definition.py

@@ -271,6 +271,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
         etcd_volume_size = mock_cluster.labels.get(
             'etcd_volume_size')
         kube_tag = mock_cluster.labels.get('kube_tag')
+        etcd_tag = mock_cluster.labels.get('etcd_tag')
+        flannel_tag = mock_cluster.labels.get('flannel_tag')
         container_infra_prefix = mock_cluster.labels.get(
             'container_infra_prefix')
         availability_zone = mock_cluster.labels.get(
@@ -312,6 +314,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
             'magnum_url': mock_osc.magnum_url.return_value,
             'region_name': mock_osc.cinder_region_name.return_value,
             'kube_tag': kube_tag,
+            'etcd_tag': etcd_tag,
+            'flannel_tag': flannel_tag,
             'container_infra_prefix': container_infra_prefix,
             'nodes_affinity_policy': 'soft-anti-affinity',
             'availability_zone': availability_zone,
@@ -379,6 +383,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
         etcd_volume_size = mock_cluster.labels.get(
             'etcd_volume_size')
         kube_tag = mock_cluster.labels.get('kube_tag')
+        etcd_tag = mock_cluster.labels.get('etcd_tag')
+        flannel_tag = mock_cluster.labels.get('flannel_tag')
         container_infra_prefix = mock_cluster.labels.get(
             'container_infra_prefix')
         availability_zone = mock_cluster.labels.get(
@@ -422,6 +428,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
             'loadbalancing_protocol': 'HTTP',
             'kubernetes_port': 8080,
             'kube_tag': kube_tag,
+            'etcd_tag': etcd_tag,
+            'flannel_tag': flannel_tag,
             'container_infra_prefix': container_infra_prefix,
             'nodes_affinity_policy': 'soft-anti-affinity',
             'availability_zone': availability_zone,

releasenotes/notes/update-to-f27-cc8aa873cdf111bc.yaml

@@ -0,0 +1,10 @@
+---
+features:
+  - |
+    Update k8s_fedora_atomic driver to the latest Fedora Atomic 27 release
+    and run etcd and flanneld in system containers which are removed from
+    the base OS.
+upgrade:
+  - |
+    New clusters should be created with kube_tag=v1.9.3 or later. v1.9.3 is
+    the default version in the queens release.