DIB elements to support dcos for magnum

This is a patch to build a centos image to support installing DC/OS as
a magnum coe.

Two elements are included: dcos and docker, only used to build a centos
image.

1. Element docker will install and configure docker in centos.
2. Element dcos will download dcos_generate_config.sh and do some
configurations for DC/OS.

Design spec in contrib/drivers/dcos_centos_v1/image/README.md.

Partially-Implements: blueprint mesos-dcos

Change-Id: I30fa4c102205aa2475b9491398c06da0d1e86f84

contrib/drivers/dcos_centos_v1/image/README.md

@@ -0,0 +1,86 @@
+=============
+centos-dcos
+=============
+
+This directory contains `[diskimage-builder](https://github.com/openstack/diskimage-builder)`
+elements to build an centos image which contains dcos.
+
+
+Pre-requisites to run diskimage-builder
+---------------------------------------
+
+For diskimage-builder to work, following packages need to be
+present:
+
+* kpartx
+* qemu-utils
+* curl
+* xfsprogs
+* yum
+* yum-utils
+* git
+
+For Debian/Ubuntu systems, use::
+
+    apt-get install kpartx qemu-utils curl xfsprogs yum yum-utils git
+
+For CentOS and Fedora < 22, use::
+
+    yum install kpartx qemu-utils curl xfsprogs yum yum-utils git
+
+For Fedora >= 22, use::
+
+    dnf install kpartx @virtualization curl xfsprogs yum yum-utils git
+
+
+How to generate Centos image with DC/OS 1.8.x
+---------------------------------------------
+
+1. Download and export element path
+
+   git clone https://git.openstack.org/openstack/magnum
+   git clone https://git.openstack.org/openstack/diskimage-builder.git
+   git clone https://git.openstack.org/openstack/dib-utils.git
+   git clone https://git.openstack.org/openstack/tripleo-image-elements.git
+   git clone https://git.openstack.org/openstack/heat-templates.git
+
+   export PATH="${PWD}/diskimage-builder/bin:$PATH"
+   export PATH="${PWD}/dib-utils/bin:$PATH"
+   export ELEMENTS_PATH=magnum/contrib/drivers/dcos_centos_v1/image
+   export ELEMENTS_PATH=${ELEMENTS_PATH}:diskimage-builder/elements
+   export ELEMENTS_PATH=${ELEMENTS_PATH}:tripleo-image-elements/elements:heat-templates/hot/software-config/elements
+
+2. Export environment path of the url to download dcos_generate_config.sh
+   This default download url is for DC/OS 1.8.4
+
+   export DCOS_GENERATE_CONFIG_SRC=https://downloads.dcos.io/dcos/stable/commit/e64024af95b62c632c90b9063ed06296fcf38ea5/dcos_generate_config.sh
+
+   Or specify local file path
+
+   export DCOS_GENERATE_CONFIG_SRC=`pwd`/dcos_generate_config.sh
+
+3. Set file system type to `xfs`
+   Only XFS is currently supported for overlay.
+   See https://dcos.io/docs/1.8/administration/installing/custom/system-requirements/install-docker-centos/#recommendations
+
+   export FS_TYPE=xfs
+
+4. Create image
+
+   disk-image-create \
+   centos7 vm docker dcos selinux-permissive \
+   os-collect-config os-refresh-config os-apply-config \
+   heat-config heat-config-script \
+   -o centos-7-dcos.qcow2
+
+5. (Optional) Create user image for bare metal node
+   Create with elements dhcp-all-interfaces and devuser
+
+   export DIB_DEV_USER_USERNAME=centos
+   export DIB_DEV_USER_PWDLESS_SUDO=YES
+
+   disk-image-create \
+   centos7 vm docker dcos selinux-permissive dhcp-all-interfaces devuser \
+   os-collect-config os-refresh-config os-apply-config \
+   heat-config heat-config-script \
+   -o centos-7-dcos-bm.qcow2

contrib/drivers/dcos_centos_v1/image/dcos/elements-deps

@@ -0,0 +1,2 @@
+package-installs
+docker

contrib/drivers/dcos_centos_v1/image/dcos/environment.d/10-dcos-install-url

@@ -0,0 +1,5 @@
+# Specify download url, default DC/OS version 1.8.4
+export DCOS_GENERATE_CONFIG_SRC=${DCOS_GENERATE_CONFIG_SRC:-https://downloads.dcos.io/dcos/stable/commit/e64024af95b62c632c90b9063ed06296fcf38ea5/dcos_generate_config.sh}
+
+# or local file path
+# export DCOS_GENERATE_CONFIG_SRC=${DCOS_GENERATE_CONFIG_SRC:-${PWD}/dcos_generate_config.sh}

contrib/drivers/dcos_centos_v1/image/dcos/extra-data.d/99-download-generate-config

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# This scrpit file is used to dowload dcos_generate_config.sh outside chroot.
+# Ihis file is essential that the size of dcos_generate_config.sh is more than
+# 700M, we should download it into the image in advance.
+
+sudo mkdir -p $TMP_MOUNT_PATH/opt/dcos
+
+if [ -f $DCOS_GENERATE_CONFIG_SRC ]; then
+    # If $DCOS_GENERATE_CONFIG_SRC is a file path, copy the file
+    sudo cp $DCOS_GENERATE_CONFIG_SRC $TMP_MOUNT_PATH/opt/dcos
+else
+    # If $DCOS_GENERATE_CONFIG_SRC is a url, download it
+    # Please make sure curl is installed on your host environment
+    cd $TMP_MOUNT_PATH/opt/dcos
+    sudo -E curl -O $DCOS_GENERATE_CONFIG_SRC
+fi

contrib/drivers/dcos_centos_v1/image/dcos/package-installs.yaml

@@ -0,0 +1,6 @@
+tar:
+xz:
+unzip:
+curl:
+ipset:
+ntp:

contrib/drivers/dcos_centos_v1/image/dcos/post-install.d/99-add-norgoup

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# nogroup will be used on Mesos masters and agents.
+sudo groupadd nogroup

contrib/drivers/dcos_centos_v1/image/dcos/post-install.d/99-disable-firewall

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# CentOS-7 introduced firewalld, which is a wrapper around iptables
+# and can conflict with Docker.
+# See https://docs.docker.com/v1.6/installation/centos/#firewalld
+sudo systemctl disable firewalld

contrib/drivers/dcos_centos_v1/image/dcos/post-install.d/99-enable-ntp

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+sudo systemctl enable ntpd

contrib/drivers/dcos_centos_v1/image/docker/elements-deps

@@ -0,0 +1 @@
+package-installs

contrib/drivers/dcos_centos_v1/image/docker/install.d/50-install-docker

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# Install the Docker engine, daemon, and service.
+#
+# The supported versions of Docker are:
+# 1.7.x
+# 1.8.x
+# 1.9.x
+# 1.10.x
+# 1.11.x
+# Docker 1.12.x is NOT supported.
+# Docker 1.9.x - 1.11.x is recommended for stability reasons.
+# https://github.com/docker/docker/issues/9718
+#
+# See DC/OS installtion guide for details
+# https://dcos.io/docs/1.8/administration/installing/custom/system-requirements/install-docker-centos/
+#
+sudo -E yum install -y docker-engine-1.11.2

contrib/drivers/dcos_centos_v1/image/docker/post-install.d/60-enable-docker-service

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+sudo systemctl enable docker

contrib/drivers/dcos_centos_v1/image/docker/pre-install.d/10-enable-overlay

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# Upgrade CentOS to 7.2
+sudo -E yum upgrade --assumeyes --tolerant
+sudo -E yum update --assumeyes
+
+# Verify that the kernel is at least 3.10
+function version_gt() { test "$(echo "$@" | tr " " "\n" | sort -V | head -n 1)" != "$1"; }
+
+kernel_version=`uname -r | cut --bytes=1-4`
+expect_version=3.10
+if version_gt $expect_version $kernel_version; then
+    echo "Error: kernel version at least $expect_version, current version $kernel_version"
+    exit 1
+fi
+
+# Enable OverlayFS
+sudo tee /etc/modules-load.d/overlay.conf <<-'EOF'
+overlay
+EOF

contrib/drivers/dcos_centos_v1/image/docker/pre-install.d/20-configure-docker-service

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+    set -x
+fi
+set -eu
+set -o pipefail
+
+# Configure yum to use the Docker yum repo
+sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
+[dockerrepo]
+name=Docker Repository
+baseurl=https://yum.dockerproject.org/repo/main/centos/7/
+enabled=1
+gpgcheck=1
+gpgkey=https://yum.dockerproject.org/gpg
+EOF
+
+# Configure systemd to run the Docker Daemon with OverlayFS
+# Manage Docker on CentOS with systemd.
+# systemd handles starting Docker on boot and restarting it when it crashes.
+#
+# Docker 1.11.x will be installed, so issue for Docker 1.12.x on Centos7
+# won't happen.
+# https://github.com/docker/docker/issues/22847
+# https://github.com/docker/docker/issues/25098
+#
+sudo mkdir -p /etc/systemd/system/docker.service.d
+sudo tee /etc/systemd/system/docker.service.d/override.conf <<- 'EOF'
+[Service]
+ExecStart=
+ExecStart=/usr/bin/docker daemon --storage-driver=overlay -H fd://
+EOF

contrib/drivers/dcos_centos_v1/image/validate_dcos_image.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+#
+# Copyright (c) 2016 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+# check that image is valid
+qemu-img check -q $1
+
+# validate estimated size
+FILESIZE=$(stat -c%s "$1")
+MIN_SIZE=1231028224 # 1.15GB
+MAX_SIZE=1335885824 # 1.25GB
+
+if [ $FILESIZE -lt $MIN_SIZE ] ; then
+    echo "Error: generated image size is lower than expected."
+    exit 1
+fi
+
+if [ $FILESIZE -gt $MAX_SIZE ] ; then
+    echo "Error: generated image size is higher than expected."
+    exit 1
+fi