blueprint cluster-installation

Add install tool

The tool aims to make the deploying of stackube as easy as possible.
User could set up a whole Stackube cluster automatically by using it.
It uses docker images provided by OpenStack Kolla Project to run a
containerized OpenStack, and uses kubeadm to deploy kubenetes, then
bootstrap the Stackube cluster.

Change-Id: I6f18cf4d1a792bc505f955937f000dc0967341ce
Implements: blueprint cluster-installation

doc/source/multi_node.rst

@@ -1,8 +1,154 @@
-Setting up a multi nodes cluster Stackube
+Setting Up A Multi-nodes Stackube (Without HA For Now)
-=====================================
+======================================================
 
 This page describes how to setup a multi-nodes cluster of Stackube.
 
-=================
+Prerequisites
-TODO
+-------------
-=================
+
+Roles
+~~~~~
+
+A stackube deployment is comprised by four kinds of nodes: control,
+network, compute, storage.
+
+-  Control
+
+   -  The control node is where openstack/kubernetes/ceph's
+      control-plane will run.
+   -  **At least one and only one node** (for now).
+   -  Minimum hardware requirements:
+
+      -  Two network interfaces
+
+         -  One is for public network connection, with a public IP.
+         -  The other one is for private network connection, with a
+            private IP and MTU >= 1600.
+
+      -  8GB main memory
+      -  50GB disk space
+
+-  Network
+
+   -  The network nodes are where neutron l3/lbaas/dhcp agents will run.
+   -  At least one node.
+   -  Minimum hardware requirements:
+
+      -  Two network interfaces
+
+         -  One is as neutron-external-interface. Public IP is not
+            needed.
+         -  The other one is for private network connection, with a
+            private IP and MTU >= 1600.
+
+      -  8GB main memory
+      -  50GB disk space
+
+-  Compute
+
+   -  The compute nodes are where your workloads will run.
+   -  At least one node.
+   -  Minimum hardware requirements:
+
+      -  One network interface
+
+         -  For private network connection, with a private IP and MTU >=
+            1600.
+
+      -  8GB main memory
+      -  50GB disk space
+
+-  Storage
+
+   -  The storage nodes are where ceph-osd(s) will run.
+   -  At least one node.
+   -  Minimum hardware requirements:
+
+      -  One network interface
+
+         -  For private network connection, with a private IP and MTU >=
+            1600.
+
+      -  8GB main memory
+      -  50GB disk space
+
+There is no conflict between any two roles. That means, all of the roles
+could be deployed on the same node(s).
+
+Host OS
+~~~~~~~
+
+For now only CentOS 7.x is supported.
+
+Public IP Pool
+~~~~~~~~~~~~~~
+
+A number of public IPs are needed.
+
+Deploy
+------
+
+All instructions below **must be done on the control node.**
+
+1. SSH To The Control Node, And Become Root
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+    sudo su -
+
+2. Enable Password-Less SSH
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The control node needs to ssh to all nodes when deploying.
+
+-  Generate SSH keys on the control node. Leave the passphrase empty:
+
+::
+
+    ssh-keygen
+
+    Generating public/private rsa key pair.
+    Enter file in which to save the key (/root/.ssh/id_rsa): 
+    Enter passphrase (empty for no passphrase): 
+    Enter same passphrase again: 
+    Your identification has been saved in /root/.ssh/id_rsa.
+    Your public key has been saved in /root/.ssh/id_rsa.pub.
+
+-  Copy the key to each node (including the control node itself):
+
+::
+
+    ssh-copy-id root@NODE_IP
+
+3. Clone Stackube Repo
+~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+    git clone https://git.openstack.org/openstack/stackube
+
+4. Edit The Config File
+~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+    cd stackube/install
+    vim config_example
+
+5. Do The Deploy
+~~~~~~~~~~~~~~~~
+
+::
+
+    bash deploy.sh config_example
+
+If failed, please **do remove** (as shown below) before deploy again.
+
+Remove
+------
+
+::
+
+    bash remove.sh config_example
+

install/README.md

@@ -0,0 +1,112 @@
+# Setting Up A Multi-nodes Stackube (Without HA For Now)
+
+This page describes how to setup a multi-nodes cluster of Stackube.
+
+## Prerequisites
+
+### Roles
+
+A stackube deployment is comprised by four kinds of nodes: control, network, compute, storage.
+
+- Control
+    - The control node is where openstack/kubernetes/ceph's control-plane will run.
+    - **At least one and only one node** (for now).
+    - Minimum hardware requirements:
+        - Two network interfaces
+            - One is for public network connection, with a public IP.
+            - The other one is for private network connection, with a private IP and MTU >= 1600.
+        - 8GB main memory
+        - 50GB disk space
+
+- Network
+    - The network nodes are where neutron l3/lbaas/dhcp agents will run.
+    - At least one node.
+    - Minimum hardware requirements:
+        - Two network interfaces
+            - One is as neutron-external-interface. Public IP is not needed.
+            - The other one is for private network connection, with a private IP and MTU >= 1600.
+        - 8GB main memory
+        - 50GB disk space
+
+- Compute
+    - The compute nodes are where your workloads will run.
+    - At least one node.
+    - Minimum hardware requirements:
+        - One network interface
+            - For private network connection, with a private IP and MTU >= 1600.
+        - 8GB main memory
+        - 50GB disk space
+
+- Storage
+    - The storage nodes are where ceph-osd(s) will run.
+    - At least one node.
+    - Minimum hardware requirements:
+        - One network interface
+            - For private network connection, with a private IP and MTU >= 1600.
+        - 8GB main memory
+        - 50GB disk space
+
+There is no conflict between any two roles. That means, all of the roles could be deployed on the same node(s).
+
+### Host OS
+For now only CentOS 7.x is supported.
+
+### Public IP Pool
+A number of public IPs are needed.
+
+
+## Deploy
+
+All instructions below **must be done on the control node.**
+
+### 1. SSH To The Control Node, And Become Root 
+```
+sudo su -
+```
+
+### 2. Enable Password-Less SSH
+
+The control node needs to ssh to all nodes when deploying.
+
+- Generate SSH keys on the control node. Leave the passphrase empty:
+
+```
+ssh-keygen
+
+Generating public/private rsa key pair.
+Enter file in which to save the key (/root/.ssh/id_rsa): 
+Enter passphrase (empty for no passphrase): 
+Enter same passphrase again: 
+Your identification has been saved in /root/.ssh/id_rsa.
+Your public key has been saved in /root/.ssh/id_rsa.pub.
+```
+
+- Copy the key to each node (including the control node itself):
+```
+ssh-copy-id root@NODE_IP
+```
+
+### 3. Clone Stackube Repo
+```
+git clone https://git.openstack.org/openstack/stackube
+```
+
+### 4. Edit The Config File
+```
+cd stackube/install
+vim config_example
+```
+
+### 5. Do The Deploy
+```
+bash deploy.sh config_example
+```
+
+If failed, please **do remove** (as shown below) before deploy again.
+
+
+
+## Remove
+```
+bash remove.sh config_example
+```

install/ceph/config_ceph/ceph-mon/ceph.conf

@@ -0,0 +1,53 @@
+[global]
+log file = /var/log/kolla/ceph/$cluster-$name.log
+log to syslog = true
+err to syslog = true
+
+fsid = __FSID__
+mon initial members = __PUBLIC_IP__
+mon host = __PUBLIC_IP__
+mon addr = __PUBLIC_IP__:6789
+
+auth cluster required = cephx
+auth service required = cephx
+auth client required = cephx
+
+# NOTE(inc0): This line will mean that if ceph was upgraded, it will run as root
+#             until contents of /var/lib/ceph are chowned to ceph user.
+#             This change was introduced in Jewel version and we should include
+#             chown operation in upgrade procedure. https://bugs.launchpad.net/kolla/+bug/1620702
+setuser match path = /var/lib/ceph/$type/$cluster-$id
+
+# Starting with the Jewel release, the ceph-osd daemon will refuse to start if the configured 
+# max object name cannot be safely stored on ext4. If the cluster is only being used with short 
+# object names (e.g., RBD only), you can continue using ext4 by setting the following configuration option:
+# Note: This may result in difficult-to-diagnose errors if you try to use RGW or other librados 
+# clients that do not properly handle or politely surface any resulting ENAMETOOLONG errors.
+osd max object name len = 256
+osd max object namespace len = 64
+
+osd journal size = 5120
+osd pool default size = 1
+osd pool default min size = 1
+osd pool default pg num = 64
+osd pool default pgp num = 64
+
+osd crush chooseleaf type = 1
+
+rbd default features = 3
+
+
+[client]
+rbd cache = false
+rbd cache size = 0
+rbd cache max dirty =0
+rbd cache target dirty = 0
+rbd cache writethrough until flush = false
+
+
+[mon]
+# NOTE(SamYaple): The monitor files have been known to grow very large. The
+#                 only fix for that is to compact the files.
+mon compact on start = true
+mon cluster log file = /var/log/kolla/ceph/$cluster.log
+

install/ceph/config_ceph/ceph-mon/config.json

@@ -0,0 +1,39 @@
+{
+    "command": "/usr/bin/ceph-mon -d -i __PUBLIC_IP__ --public-addr __PUBLIC_IP__:6789",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/ceph.conf",
+            "dest": "/etc/ceph/ceph.conf",
+            "owner": "ceph",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.client.admin.keyring",
+            "dest": "/etc/ceph/ceph.client.admin.keyring",
+            "owner": "ceph",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.client.mon.keyring",
+            "dest": "/etc/ceph/ceph.client.mon.keyring",
+            "owner": "ceph",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.client.radosgw.keyring",
+            "dest": "/etc/ceph/ceph.client.radosgw.keyring",
+            "owner": "ceph",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.monmap",
+            "dest": "/etc/ceph/ceph.monmap",
+            "owner": "ceph",
+            "perm": "0600",
+            "optional": true
+        }
+    ]
+}

install/ceph/config_ceph/ceph-osd/add_osd.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+
+set -x
+
+cp --remove-destination /var/lib/kolla/config_files/{ceph.client.admin.keyring,ceph.conf} /etc/ceph/ || exit 1
+
+ceph osd crush add-bucket __PUBLIC_IP__ host || exit 1
+ceph osd crush move __PUBLIC_IP__ root=default || exit 1
+
+num=`ceph osd create` || exit 1
+echo $num || exit 1
+mkdir -p /var/lib/ceph/osd/ceph-${num} || exit 1
+ceph-osd -i ${num} --mkfs --mkkey || exit 1
+ceph auth add osd.${num} osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-${num}/keyring || exit 1
+ceph osd crush add osd.${num} 1.0 host=__PUBLIC_IP__ || exit 1
+
+chown ceph:ceph /var/lib/ceph/osd -R  || exit 1
+
+ceph osd crush tree
+
+exit 0
+

install/ceph/config_ceph/ceph-osd/config.json

@@ -0,0 +1,17 @@
+{ 
+    "command": "/usr/bin/ceph-osd -f -d --public-addr __PUBLIC_IP__ --cluster-addr __CLUSTER_IP__",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/ceph.conf",
+            "dest": "/etc/ceph/ceph.conf",
+            "owner": "ceph",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.client.admin.keyring",
+            "dest": "/etc/ceph/ceph.client.admin.keyring",
+            "owner": "ceph",
+            "perm": "0600"
+        }
+    ]
+}

install/ceph/deploy_ceph_mon.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``CEPH_MON_PUBLIC_IP``
+# - ``CEPH_FSID``  must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/ceph
+chmod 777 /var/log/stackube/ceph
+
+
+## config files
+mkdir -p /etc/stackube/ceph
+cp -a ${programDir}/config_ceph/ceph-mon /etc/stackube/ceph/
+sed -i "s/__FSID__/${CEPH_FSID}/g" /etc/stackube/ceph/ceph-mon/ceph.conf
+sed -i "s/__PUBLIC_IP__/${CEPH_MON_PUBLIC_IP}/g" /etc/stackube/ceph/ceph-mon/ceph.conf
+sed -i "s/__PUBLIC_IP__/${CEPH_MON_PUBLIC_IP}/g" /etc/stackube/ceph/ceph-mon/config.json
+
+
+mkdir -p /var/lib/stackube/ceph/ceph_mon_config  && \
+mkdir -p /var/lib/stackube/ceph/ceph_mon  && \
+docker run --net host  \
+    --name stackube_ceph_bootstrap_mon  \
+    -v /etc/stackube/ceph/ceph-mon/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/ceph:/var/log/kolla/:rw  \
+    -v /var/lib/stackube/ceph/ceph_mon_config:/etc/ceph/:rw  \
+    -v /var/lib/stackube/ceph/ceph_mon:/var/lib/ceph/:rw  \
+    \
+    -e "KOLLA_BOOTSTRAP="  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    -e "MON_IP=${CEPH_MON_PUBLIC_IP}" \
+    -e "HOSTNAME=${CEPH_MON_PUBLIC_IP}" \
+    kolla/centos-binary-ceph-mon:4.0.0
+
+docker rm stackube_ceph_bootstrap_mon
+
+
+docker run -d  --net host  \
+    --name stackube_ceph_mon  \
+    -v /etc/stackube/ceph/ceph-mon/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/ceph:/var/log/kolla/:rw  \
+    -v /var/lib/stackube/ceph/ceph_mon_config:/etc/ceph/:rw  \
+    -v /var/lib/stackube/ceph/ceph_mon:/var/lib/ceph/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=ceph-mon"  \
+    -e "HOSTNAME=${CEPH_MON_PUBLIC_IP}"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-ceph-mon:4.0.0
+
+sleep 5
+
+docker exec stackube_ceph_mon ceph -s
+
+
+
+exit 0

install/ceph/deploy_ceph_osd.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``CEPH_OSD_PUBLIC_IP``, ``CEPH_OSD_CLUSTER_IP``,
+# - ``CEPH_OSD_DATA_DIR``   must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/ceph
+chmod 777 /var/log/stackube/ceph
+
+
+## config files
+sed -i "s/__PUBLIC_IP__/${CEPH_OSD_PUBLIC_IP}/g" /etc/stackube/ceph/ceph-osd/add_osd.sh
+sed -i "s/__PUBLIC_IP__/${CEPH_OSD_PUBLIC_IP}/g" /etc/stackube/ceph/ceph-osd/config.json
+sed -i "s/__CLUSTER_IP__/${CEPH_OSD_CLUSTER_IP}/g" /etc/stackube/ceph/ceph-osd/config.json
+
+
+## bootstrap
+mkdir -p ${CEPH_OSD_DATA_DIR}
+docker run --net host  \
+    --name stackube_ceph_bootstrap_osd  \
+    -v /etc/stackube/ceph/ceph-osd/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/ceph:/var/log/kolla/:rw  \
+    -v ${CEPH_OSD_DATA_DIR}:/var/lib/ceph/:rw  \
+    \
+    kolla/centos-binary-ceph-osd:4.0.0 /bin/bash /var/lib/kolla/config_files/add_osd.sh 
+
+docker rm stackube_ceph_bootstrap_osd
+
+
+## run
+theOsd=`ls ${CEPH_OSD_DATA_DIR}/osd/ | grep -- 'ceph-' | head -n 1`
+[ "${theOsd}" ]
+osdId=`echo $theOsd | awk -F\- '{print $NF}'`
+[ "${osdId}" ]
+
+docker run -d  --net host  \
+    --name stackube_ceph_osd_${osdId}  \
+    -v /etc/stackube/ceph/ceph-osd/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/ceph:/var/log/kolla/:rw  \
+    -v ${CEPH_OSD_DATA_DIR}:/var/lib/ceph/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=ceph-osd"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    -e "OSD_ID=${osdId}"  \
+    -e "JOURNAL_PARTITION=/var/lib/ceph/osd/ceph-${osdId}/journal" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-ceph-osd:4.0.0
+
+sleep 5
+
+
+
+exit 0

install/ceph/remove_ceph_from_node.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -x
+
+
+## remove docker containers
+stackubeCephConstaners=`docker ps -a | awk '{print $NF}' | grep '^stackube_ceph_' `
+if [ "${stackubeCephConstaners}" ]; then
+    docker rm -f $stackubeCephConstaners || exit 1
+fi
+
+## rm dirs
+rm -fr /etc/stackube/ceph  /var/log/stackube/ceph  /var/lib/stackube/ceph  ${CEPH_OSD_DATA_DIR} || exit 1
+
+
+
+exit 0
+

install/config_example

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+
+# Control Node
+# At least one and only one node (for now).
+CONTROL_NODE_PUBLIC_IP='147.75.77.67'
+CONTROL_NODE_PRIVATE_IP='192.168.1.1'
+
+# Network Node(s)
+# At least one node. Could be more (separated by commas).
+NETWORK_NODES_PRIVATE_IP='192.168.1.2'
+NETWORK_NODES_NEUTRON_EXT_IF='eth2'
+
+# Compute Node(s)
+# At least one node. Could be more (separated by commas).
+COMPUTE_NODES_PRIVATE_IP='192.168.1.1,192.168.1.2'
+
+# Storage Node(s)
+# At least one node. Could be more (separated by commas).
+STORAGE_NODES_PRIVATE_IP='192.168.1.1,192.168.1.2'
+STORAGE_NODES_CEPH_OSD_DATA_DIR='/var/lib/stackube/ceph/ceph_osd,/var/lib/stackube/ceph/ceph_osd'
+
+
+# NEUTRON_PUBLIC_SUBNET='subnet-range;gateway;allocation-pool'
+NEUTRON_PUBLIC_SUBNET='147.75.192.224/29;147.75.192.225;start=147.75.192.226,end=147.75.192.230'
+
+
+#CEPH_FSID=
+
+#RABBITMQ_PWD=
+
+#MYSQL_ROOT_PWD=
+#MYSQL_KEYSTONE_PWD=
+#MYSQL_NEUTRON_PWD=
+#MYSQL_CINDER_PWD=
+
+#KEYSTONE_ADMIN_PWD=
+#KEYSTONE_NEUTRON_PWD=
+#KEYSTONE_CINDER_PWD=
+
+

install/deploy.sh

@@ -0,0 +1,123 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+
+function usage {
+    echo "
+Usage:
+   bash $(basename $0) CONFIG_FILE
+"
+}
+
+[ "$1" ] || { usage; exit 1; }
+[ -f "$1" ] || { echo "Error: $1 not exists or not a file!"; exit 1; }
+
+source $(readlink -f $1) || { echo "'source $(readlink -f $1)' failed!"; exit 1; }
+
+[ "${CONTROL_NODE_PUBLIC_IP}" ] || { echo "Error: CONTROL_NODE_PUBLIC_IP not defined!"; exit 1; }
+[ "${CONTROL_NODE_PRIVATE_IP}" ] || { echo "Error: CONTROL_NODE_PRIVATE_IP not defined!"; exit 1; }
+
+[ "${NETWORK_NODES_PRIVATE_IP}" ] || { echo "Error: NETWORK_NODES_PRIVATE_IP not defined!"; exit 1; }
+[ "${NETWORK_NODES_NEUTRON_EXT_IF}" ] || { echo "Error: NETWORK_NODES_NEUTRON_EXT_IF not defined!"; exit 1; }
+
+[ "${COMPUTE_NODES_PRIVATE_IP}" ] || { echo "Error: COMPUTE_NODES_PRIVATE_IP not defined!"; exit 1; }
+
+[ "${STORAGE_NODES_PRIVATE_IP}" ] || { echo "Error: STORAGE_NODES_PRIVATE_IP not defined!"; exit 1; }
+[ "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" ] || { echo "Error: STORAGE_NODES_CEPH_OSD_DATA_DIR not defined!"; exit 1; }
+
+[ "${NEUTRON_PUBLIC_SUBNET}" ] || { echo "Error: NEUTRON_PUBLIC_SUBNET not defined!"; exit 1; }
+
+
+#####################
+
+
+function all_nodes_check_distro {
+    for IP in $1; do
+        ssh root@${IP} 'mkdir -p /tmp/stackube_install' 
+        scp ${programDir}/{ensure_distro_supported.sh,lib_common.sh} root@${IP}:/tmp/stackube_install/
+        ssh root@${IP} "/bin/bash /tmp/stackube_install/ensure_distro_supported.sh"
+    done
+}
+
+function all_nodes_install_docker {
+    for IP in $1; do
+        ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+        scp ${programDir}/ensure_docker_installed.sh root@${IP}:/tmp/stackube_install/
+        ssh root@${IP} "/bin/bash /tmp/stackube_install/ensure_docker_installed.sh"
+    done
+}
+
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log
+logDir='/var/log/stackube'
+logFile="${logDir}/install.log-$(date '+%Y-%m-%d_%H-%M-%S')"
+mkdir -p ${logDir}
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${NETWORK_NODES_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}
+${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+{
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') all_nodes_check_distro"
+    all_nodes_check_distro "${allIpList}"
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') all_nodes_install_docker"
+    all_nodes_install_docker "${allIpList}"
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') deploy_ceph"
+    bash ${programDir}/deploy_ceph.sh $(readlink -f $1)
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') deploy_openstack"
+    bash ${programDir}/deploy_openstack.sh $(readlink -f $1)
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') deploy_kubernetes"
+    bash ${programDir}/deploy_kubernetes.sh $(readlink -f $1)
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') All done!"
+
+    echo "
+Additional information:
+ * File /etc/stackube/openstack/admin-openrc.sh has been created. To use openstack command line tools you need to source the file.
+ * File /etc/kubernetes/admin.conf has been created. To use kubectl you need to do 'export KUBECONFIG=/etc/kubernetes/admin.conf'.
+ * The installation log file is available at: ${logFile}
+"
+
+} 2>&1 | tee -a ${logFile}
+
+
+
+
+exit 0
+
+
+
+
+
+
+

install/deploy_ceph.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" ]
+
+
+# ceph-mon
+export CEPH_MON_PUBLIC_IP="${CONTROL_NODE_PRIVATE_IP}"
+export CEPH_FSID=${CEPH_FSID:-aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee}
+/bin/bash ${programDir}/ceph/deploy_ceph_mon.sh
+
+
+# ceph-osd
+storageIpList=(`echo "${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g'`)
+osdDataDirList=(`echo "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" | sed -e 's/,/\n/g'`)
+[ ${#storageIpList[@]} -eq ${#osdDataDirList[@]} ]
+
+MAX=$((${#storageIpList[@]} - 1))
+for i in `seq 0 ${MAX}`; do
+    IP="${storageIpList[$i]}"
+    dataDir="${osdDataDirList[$i]}"
+    echo -e "\n------ ${IP} ${dataDir} ------"
+    ssh root@${IP} 'mkdir -p /etc/stackube/ceph /tmp/stackube_install'
+    scp -r ${programDir}/ceph/config_ceph/ceph-osd root@${IP}:/etc/stackube/ceph/
+    scp -r /var/lib/stackube/ceph/ceph_mon_config/{ceph.client.admin.keyring,ceph.conf} root@${IP}:/etc/stackube/ceph/ceph-osd/
+
+    scp ${programDir}/ceph/deploy_ceph_osd.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export CEPH_OSD_PUBLIC_IP='${IP}'
+                    export CEPH_OSD_CLUSTER_IP='${IP}'
+                    export CEPH_OSD_DATA_DIR='${dataDir}'
+                    /bin/bash /tmp/stackube_install/deploy_ceph_osd.sh"
+done
+
+docker exec stackube_ceph_mon ceph -s
+
+
+

install/deploy_kubernetes.sh

@@ -0,0 +1,148 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PUBLIC_IP}" ]
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+[ "${NETWORK_NODES_PRIVATE_IP}" ]
+[ "${COMPUTE_NODES_PRIVATE_IP}" ]
+
+
+export KUBERNETES_API_PUBLIC_IP="${CONTROL_NODE_PUBLIC_IP}"
+export KUBERNETES_API_PRIVATE_IP="${CONTROL_NODE_PRIVATE_IP}"
+export KEYSTONE_URL="https://${CONTROL_NODE_PRIVATE_IP}:5001/v2.0"
+export KEYSTONE_ADMIN_URL="https://${CONTROL_NODE_PRIVATE_IP}:35358/v2.0"
+export CLUSTER_CIDR="10.244.0.0/16"
+export CLUSTER_GATEWAY="10.244.0.1"
+export CONTAINER_CIDR="10.244.1.0/24"
+export FRAKTI_VERSION="v1.0"
+
+
+########## control & compute nodes ##########
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+# hyperd frakti
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/kubernetes/deploy_hyperd_frakti.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export FRAKTI_VERSION='${FRAKTI_VERSION}'
+                    export STREAMING_SERVER_ADDR='${IP}'
+                    /bin/bash /tmp/stackube_install/deploy_hyperd_frakti.sh"
+done
+
+# kubeadm kubectl kubelet
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/kubernetes/deploy_kubeadm_kubectl_kubelet.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/deploy_kubeadm_kubectl_kubelet.sh"
+done
+
+
+
+########## control node ##########
+
+# kubernetes master
+sed -i "s|__KEYSTONE_URL__|${KEYSTONE_URL}|g" ${programDir}/kubernetes/kubeadm.yaml
+sed -i "s|__POD_NET_CIDR__|${CLUSTER_CIDR}|g" ${programDir}/kubernetes/kubeadm.yaml
+sed -i "s/__KUBERNETES_API_PUBLIC_IP__/${KUBERNETES_API_PUBLIC_IP}/g" ${programDir}/kubernetes/kubeadm.yaml
+sed -i "s/__KUBERNETES_API_PRIVATE_IP__/${KUBERNETES_API_PRIVATE_IP}/g" ${programDir}/kubernetes/kubeadm.yaml
+/bin/bash ${programDir}/kubernetes/deploy_kubernetes_init_master.sh
+sleep 3
+
+
+
+export KUBECONFIG=/etc/kubernetes/admin.conf
+
+
+# install stackube addons
+/bin/bash ${programDir}/kubernetes/deploy_kubernetes_install_stackube_addons.sh
+sleep 10
+
+
+# add nodes
+KUBEADM_TOKEN=`kubeadm token list | grep 'kubeadm init' | head -1 | awk '{print $1}'`
+allIpList=`echo "
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq | grep -v "${CONTROL_NODE_PRIVATE_IP}"`
+for IP in ${allIpList}; do
+    ssh root@${IP} "kubeadm join --token ${KUBEADM_TOKEN} ${CONTROL_NODE_PRIVATE_IP}:6443"
+done
+
+
+# Enable schedule pods on the master (control node) if it's also designated as a compute node
+set +e
+check=`echo "
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq | grep "${CONTROL_NODE_PRIVATE_IP}" `
+if [ "${check}" ]; then
+    kubectl taint nodes $(hostname) node-role.kubernetes.io/master-
+fi
+set -e
+
+
+# certificate approve
+sleep 5
+/bin/bash ${programDir}/kubernetes/deploy_kubernetes_certificate_approve.sh
+
+
+
+## check
+sleep 3
+kubectl get nodes
+kubectl get csr --all-namespaces
+
+
+
+
+########## control (k8s master) & compute nodes ###########
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+# install ovs for cni
+for IP in ${allIpList}; do
+    ssh root@${IP} "yum install centos-release-openstack-ocata.noarch -y"
+    ssh root@${IP} "yum install openvswitch -y"
+done
+
+# install ceph for kubelet
+for IP in ${allIpList}; do
+    ssh root@${IP} "yum install centos-release-openstack-ocata.noarch -y"
+    ssh root@${IP} "yum install ceph -y"
+    ssh root@${IP} "systemctl disable ceph.target ceph-mds.target ceph-mon.target ceph-osd.target"
+    scp -r /var/lib/stackube/ceph/ceph_mon_config/*  root@${IP}:/etc/ceph/
+    ssh root@${IP} "ceph -s"
+    ssh root@${IP} "rbd -p cinder --id cinder --keyring=/etc/ceph/ceph.client.cinder.keyring ls"
+done
+
+
+
+
+exit 0

install/deploy_openstack.sh

@@ -0,0 +1,280 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+
+[ "${NETWORK_NODES_PRIVATE_IP}" ]
+#[ "${NETWORK_NODES_NEUTRON_EXT_IF}" ]
+
+[ "${NEUTRON_PUBLIC_SUBNET}" ]
+
+[ "${COMPUTE_NODES_PRIVATE_IP}" ]
+
+[ "${STORAGE_NODES_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" ]
+
+
+export OPENSTACK_ENDPOINT_IP="${CONTROL_NODE_PRIVATE_IP}"
+export KEYSTONE_API_IP="${CONTROL_NODE_PRIVATE_IP}"
+export NEUTRON_API_IP="${CONTROL_NODE_PRIVATE_IP}"
+export CINDER_API_IP="${CONTROL_NODE_PRIVATE_IP}"
+
+export MYSQL_HOST="${CONTROL_NODE_PRIVATE_IP}"
+export MYSQL_ROOT_PWD=${MYSQL_ROOT_PWD:-MysqlRoot123}
+export MYSQL_KEYSTONE_PWD=${MYSQL_KEYSTONE_PWD:-MysqlKeystone123}
+export MYSQL_NEUTRON_PWD=${MYSQL_NEUTRON_PWD:-MysqlNeutron123}
+export MYSQL_CINDER_PWD=${MYSQL_CINDER_PWD:-MysqlCinder123}
+
+export RABBITMQ_HOST="${CONTROL_NODE_PRIVATE_IP}"
+export RABBITMQ_PWD=${RABBITMQ_PWD:-rabbitmq123}
+
+export KEYSTONE_ADMIN_PWD=${KEYSTONE_ADMIN_PWD:-KeystoneAdmin123}
+export KEYSTONE_NEUTRON_PWD=${KEYSTONE_NEUTRON_PWD:-KeystoneNeutron123}
+export KEYSTONE_CINDER_PWD=${KEYSTONE_CINDER_PWD:-KeystoneCinder123}
+
+
+
+
+########## all nodes ##########
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${NETWORK_NODES_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}
+${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+# kolla-toolbox
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/kolla-toolbox root@${IP}:/etc/stackube/openstack/
+
+    scp ${programDir}/openstack/deploy_openstack_kolla_toolbox.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/deploy_openstack_kolla_toolbox.sh"
+done
+
+
+
+########## control node ##########
+
+# db, mq, haproxy
+/bin/bash ${programDir}/openstack/deploy_openstack_mariadb.sh
+/bin/bash ${programDir}/openstack/deploy_openstack_rabbitmq.sh
+/bin/bash ${programDir}/openstack/deploy_openstack_haproxy.sh
+
+# keystone
+/bin/bash ${programDir}/openstack/deploy_openstack_keystone.sh
+
+
+# neutron server
+function process_neutron_conf {
+    local configFile="$1"
+    sed -i "s/__RABBITMQ_HOST__/${RABBITMQ_HOST}/g" ${configFile}
+    sed -i "s/__RABBITMQ_PWD__/${RABBITMQ_PWD}/g" ${configFile}
+    sed -i "s/__NEUTRON_API_IP__/${NEUTRON_API_IP}/g" ${configFile}
+    sed -i "s/__MYSQL_HOST__/${MYSQL_HOST}/g" ${configFile}
+    sed -i "s/__OPENSTACK_ENDPOINT_IP__/${OPENSTACK_ENDPOINT_IP}/g" ${configFile}
+    sed -i "s/__KEYSTONE_NEUTRON_PWD__/${KEYSTONE_NEUTRON_PWD}/g" ${configFile}
+    sed -i "s/__MYSQL_NEUTRON_PWD__/${MYSQL_NEUTRON_PWD}/g" ${configFile}
+}
+
+mkdir -p /etc/stackube/openstack
+cp -a ${programDir}/openstack/config_openstack/neutron-server /etc/stackube/openstack/
+process_neutron_conf /etc/stackube/openstack/neutron-server/neutron.conf
+
+source /etc/stackube/openstack/admin-openrc.sh 
+cp -f ${OS_CACERT} /etc/stackube/openstack/neutron-server/haproxy-ca.crt
+
+/bin/bash ${programDir}/openstack/deploy_openstack_neutron_server.sh
+
+
+## cinder api
+function process_cinder_conf {
+    local cinderConfigFile="$1"
+    sed -i "s/__CINDER_API_IP__/${CINDER_API_IP}/g" ${cinderConfigFile}
+    sed -i "s/__RABBITMQ_HOST__/${RABBITMQ_HOST}/g" ${cinderConfigFile}
+    sed -i "s/__RABBITMQ_PWD__/${RABBITMQ_PWD}/g" ${cinderConfigFile}
+    sed -i "s/__MYSQL_CINDER_PWD__/${MYSQL_CINDER_PWD}/g" ${cinderConfigFile}
+    sed -i "s/__MYSQL_HOST__/${MYSQL_HOST}/g" ${cinderConfigFile}
+    sed -i "s/__OPENSTACK_ENDPOINT_IP__/${OPENSTACK_ENDPOINT_IP}/g" ${cinderConfigFile}
+    sed -i "s/__KEYSTONE_CINDER_PWD__/${KEYSTONE_CINDER_PWD}/g" ${cinderConfigFile}
+}
+mkdir -p /etc/stackube/openstack
+cp -a ${programDir}/openstack/config_openstack/cinder-api /etc/stackube/openstack/
+process_cinder_conf /etc/stackube/openstack/cinder-api/cinder.conf
+
+source /etc/stackube/openstack/admin-openrc.sh 
+cp -f ${OS_CACERT} /etc/stackube/openstack/cinder-api/haproxy-ca.crt
+
+/bin/bash ${programDir}/openstack/deploy_openstack_cinder_api.sh
+
+
+# cinder scheduler
+mkdir -p /etc/stackube/openstack
+cp -a ${programDir}/openstack/config_openstack/cinder-scheduler /etc/stackube/openstack/
+cp -f /etc/stackube/openstack/cinder-api/cinder.conf  /etc/stackube/openstack/cinder-scheduler/
+/bin/bash ${programDir}/openstack/deploy_openstack_cinder_scheduler.sh
+
+
+# cinder volume
+docker exec stackube_ceph_mon ceph osd pool create cinder 128 128
+docker exec stackube_ceph_mon ceph auth get-or-create client.cinder mon 'allow r' \
+                 osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder'
+docker exec stackube_ceph_mon /bin/bash -c 'ceph auth get-or-create client.cinder | tee /etc/ceph/ceph.client.cinder.keyring'
+
+for IP in ${CONTROL_NODE_PRIVATE_IP} ; do 
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/cinder-volume root@${IP}:/etc/stackube/openstack/
+    scp -r /etc/stackube/openstack/cinder-api/cinder.conf \
+           /var/lib/stackube/ceph/ceph_mon_config/{ceph.conf,ceph.client.cinder.keyring}  root@${IP}:/etc/stackube/openstack/cinder-volume/
+
+    scp ${programDir}/openstack/deploy_openstack_cinder_volume.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/deploy_openstack_cinder_volume.sh"
+done
+
+
+
+
+########## network nodes ##########
+
+# neutron l3_agent
+for IP in `echo ${NETWORK_NODES_PRIVATE_IP} | sed -e 's/,/ /g' ` ; do 
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/neutron-l3-agent root@${IP}:/etc/stackube/openstack/
+    scp -r /etc/stackube/openstack/neutron-server/neutron.conf \
+           ${programDir}/openstack/config_openstack/neutron-server/ml2_conf.ini  root@${IP}:/etc/stackube/openstack/neutron-l3-agent/
+
+    scp ${programDir}/openstack/deploy_openstack_neutron_l3_agent.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export OVSDB_IP='${IP}'
+                    export ML2_LOCAL_IP='${IP}'
+                    /bin/bash /tmp/stackube_install/deploy_openstack_neutron_l3_agent.sh"
+done
+
+
+# neutron dhcp_agent
+for IP in `echo ${NETWORK_NODES_PRIVATE_IP} | sed -e 's/,/ /g' ` ; do 
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/neutron-dhcp-agent root@${IP}:/etc/stackube/openstack/
+    scp -r /etc/stackube/openstack/neutron-server/neutron.conf \
+           ${programDir}/openstack/config_openstack/neutron-server/ml2_conf.ini  root@${IP}:/etc/stackube/openstack/neutron-dhcp-agent/
+
+    scp ${programDir}/openstack/deploy_openstack_neutron_dhcp_agent.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export OVSDB_IP='${IP}'
+                    export ML2_LOCAL_IP='${IP}'
+                    /bin/bash /tmp/stackube_install/deploy_openstack_neutron_dhcp_agent.sh"
+done
+
+
+# neutron lbaas_agent
+for IP in `echo ${NETWORK_NODES_PRIVATE_IP} | sed -e 's/,/ /g' ` ; do 
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/neutron-lbaas-agent root@${IP}:/etc/stackube/openstack/
+    scp -r /etc/stackube/openstack/neutron-server/neutron.conf \
+           ${programDir}/openstack/config_openstack/neutron-server/{ml2_conf.ini,neutron_lbaas.conf}  root@${IP}:/etc/stackube/openstack/neutron-lbaas-agent/
+
+    scp ${programDir}/openstack/deploy_openstack_neutron_lbaas_agent.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export OVSDB_IP='${IP}'
+                    export ML2_LOCAL_IP='${IP}'
+                    export KEYSTONE_API_IP='${KEYSTONE_API_IP}'
+                    export KEYSTONE_NEUTRON_PWD='${KEYSTONE_NEUTRON_PWD}'
+                    /bin/bash /tmp/stackube_install/deploy_openstack_neutron_lbaas_agent.sh"
+done
+
+
+
+
+
+########## control & network & compute nodes ##########
+
+# openvswitch agent (deploy on control node for k8s master)
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${NETWORK_NODES_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /etc/stackube/openstack /tmp/stackube_install'
+    scp -r ${programDir}/openstack/config_openstack/{openvswitch-db-server,openvswitch-vswitchd,neutron-openvswitch-agent} root@${IP}:/etc/stackube/openstack/
+    scp -r /etc/stackube/openstack/neutron-server/neutron.conf ${programDir}/openstack/config_openstack/neutron-server/ml2_conf.ini  root@${IP}:/etc/stackube/openstack/neutron-openvswitch-agent/
+
+    scp ${programDir}/openstack/deploy_openstack_neutron_openvswitch_agent.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export OVSDB_IP='${IP}'
+                    export ML2_LOCAL_IP='${IP}'
+                    /bin/bash /tmp/stackube_install/deploy_openstack_neutron_openvswitch_agent.sh"
+done
+
+# network nodes: NEUTRON_EXT_IF
+networkIpList=(`echo "${NETWORK_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g'`)
+neutronExtIfList=(`echo "${NETWORK_NODES_NEUTRON_EXT_IF}" | sed -e 's/,/\n/g'`)
+[ ${#networkIpList[@]} -eq ${#neutronExtIfList[@]} ]
+MAX=$((${#networkIpList[@]} - 1))
+for i in `seq 0 ${MAX}`; do
+    IP="${networkIpList[$i]}"
+    extIf="${neutronExtIfList[$i]}"
+    echo -e "\n------ ${IP} ${extIf} ------"
+    ssh root@${IP} "docker exec stackube_openstack_openvswitch_db /usr/local/bin/kolla_ensure_openvswitch_configured br-ex ${extIf}"
+done
+
+
+
+
+
+######### compute node ############
+
+# certificate for kubestack
+allIpList=`echo "
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+for IP in ${allIpList}; do
+    scp -r /etc/stackube/openstack/certificates/CA/int-ca/ca-chain.pem root@${IP}:/usr/share/pki/ca-trust-source/anchors/stackube-chain.pem
+    ssh root@${IP} "update-ca-trust"
+done
+
+
+
+
+
+######### control node ############
+
+# create public network and subnet
+yum install centos-release-openstack-ocata.noarch -y
+yum install python-openstackclient -y
+
+source /etc/stackube/openstack/admin-openrc.sh
+openstack network create --external --provider-physical-network physnet1 --provider-network-type flat public_1
+
+# NEUTRON_PUBLIC_SUBNET='subnet-range;gateway;allocation-pool'
+SUBNET=`echo "${NEUTRON_PUBLIC_SUBNET}" | awk -F\; '{print $1}'`
+GATEWAY=`echo "${NEUTRON_PUBLIC_SUBNET}" | awk -F\; '{print $2}'`
+POOL=`echo "${NEUTRON_PUBLIC_SUBNET}" | awk -F\; '{print $3}'`
+openstack subnet create  public_1-subnet_1  \
+    --subnet-range "${SUBNET}"  --gateway "${GATEWAY}"  --allocation-pool "${POOL}"  --no-dhcp  --network public_1
+
+
+# check
+openstack network list
+openstack subnet list
+openstack endpoint list

install/ensure_distro_supported.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+
+source ${programDir}/lib_common.sh || { echo "Error: 'source ${programDir}/lib_common.sh' failed!"; exit 1; }
+
+MSG='Sorry, only CentOS 7.x supported for now.'
+
+if ! is_fedora; then
+    echo ${MSG}
+    exit 1
+fi
+
+mainVersion=`echo ${os_RELEASE} | awk -F\. '{print $1}' `
+if [ "${os_VENDOR}" == "CentOS" ] && [ "${mainVersion}" == "7" ]; then
+    true
+else
+    echo ${MSG}
+    exit 1
+fi
+
+
+exit 0
+

install/ensure_docker_installed.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+set -x
+
+systemctl start docker &> /dev/null
+
+sleep 2
+
+docker info &> /dev/null
+
+if [ "$?" != "0" ]; then 
+    cat > /etc/yum.repos.d/docker.repo  << EOF
+[docker-repo]
+name=Docker main Repository
+baseurl=https://yum.dockerproject.org/repo/main/centos/7
+enabled=1
+gpgcheck=1
+gpgkey=https://yum.dockerproject.org/gpg
+EOF
+    yum install docker-engine-1.12.6 docker-engine-selinux-1.12.6 -y || exit 1
+    #sed -i 's|ExecStart=.*|ExecStart=/usr/bin/dockerd  --storage-opt dm.mountopt=nodiscard --storage-opt dm.blkdiscard=false|g' /usr/lib/systemd/system/docker.service
+    sed -i 's|ExecStart=.*|ExecStart=/usr/bin/dockerd  -s overlay |g' /usr/lib/systemd/system/docker.service
+    systemctl daemon-reload  || exit 1
+    systemctl enable docker || exit 1
+    systemctl start  docker || exit 1
+fi
+
+sleep 5
+
+docker info &> /dev/null || exit 1
+
+
+exit 0
+

install/kubernetes/deploy_hyperd_frakti.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``STREAMING_SERVER_ADDR``
+# - ``FRAKTI_VERSION``  must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## install libvirtd
+yum install -y libvirt
+
+
+## install hyperd
+CENTOS7_QEMU_HYPER="http://hypercontainer-install.s3.amazonaws.com/qemu-hyper-2.4.1-3.el7.centos.x86_64.rpm"
+CENTOS7_HYPERSTART="https://s3-us-west-1.amazonaws.com/hypercontainer-build/1.0-rc2/centos/hyperstart-0.8.1-1.el7.centos.x86_64.rpm"
+CENTOS7_HYPER="https://s3-us-west-1.amazonaws.com/hypercontainer-build/1.0-rc2/centos/hyper-container-0.8.1-1.el7.centos.x86_64.rpm"
+
+if rpm -qa | grep "hyper-container-0.8.1-1.el7.centos.x86_64" ; then
+    true
+else
+    set -e
+    yum install -y ${CENTOS7_QEMU_HYPER} ${CENTOS7_HYPERSTART} ${CENTOS7_HYPER}
+    set +e
+fi
+set -e
+
+cat > /etc/hyper/config << EOF
+Kernel=/var/lib/hyper/kernel
+Initrd=/var/lib/hyper/hyper-initrd.img
+Hypervisor=qemu
+StorageDriver=overlay
+gRPCHost=127.0.0.1:22318
+
+EOF
+
+
+## install frakti
+set +e
+[ -f /usr/bin/frakti ] && rm -f /usr/bin/frakti
+set -e
+curl -sSL https://github.com/kubernetes/frakti/releases/download/${FRAKTI_VERSION}/frakti -o /usr/bin/frakti 
+chmod +x /usr/bin/frakti
+
+dockerInfo=`docker info `
+cgroup_driver=`echo "${dockerInfo}" | awk '/Cgroup Driver/{print $3}' `
+[ "${cgroup_driver}" ]
+
+echo "[Unit]
+Description=Hypervisor-based container runtime for Kubernetes
+Documentation=https://github.com/kubernetes/frakti
+After=network.target
+[Service]
+ExecStart=/usr/bin/frakti --v=3 \
+          --log-dir=/var/log/frakti \
+          --logtostderr=false \
+          --cgroup-driver=${cgroup_driver} \
+          --listen=/var/run/frakti.sock \
+          --streaming-server-addr=${STREAMING_SERVER_ADDR} \
+          --hyper-endpoint=127.0.0.1:22318
+MountFlags=shared
+#TasksMax=8192
+LimitNOFILE=1048576
+LimitNPROC=1048576
+LimitCORE=infinity
+TimeoutStartSec=0
+Restart=on-abnormal
+[Install]
+WantedBy=multi-user.target
+"  > /lib/systemd/system/frakti.service 
+
+
+## start services
+systemctl daemon-reload
+systemctl enable hyperd frakti libvirtd
+systemctl restart hyperd libvirtd
+sleep 5
+systemctl restart frakti
+sleep 5
+
+## check
+hyperctl list 
+pgrep -f '/usr/bin/frakti' 
+[ -e /var/run/frakti.sock ] 
+
+
+
+exit 0

install/kubernetes/deploy_kubeadm_kubectl_kubelet.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+
+setenforce 0
+sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
+
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## install kubeadm kubectl kubelet
+cat > /etc/yum.repos.d/kubernetes.repo << EOF
+[kubernetes]
+name=Kubernetes
+baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
+enabled=1
+gpgcheck=1
+repo_gpgcheck=1
+gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
+       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
+EOF
+
+yum install -y kubelet-1.7.4-0 kubeadm-1.7.4-0 kubectl-1.7.4-0
+
+# configure_kubelet
+unitFile='/etc/systemd/system/kubelet.service.d/10-kubeadm.conf'
+sed -i '/^Environment="KUBELET_EXTRA_ARGS=/d'  ${unitFile} 
+sed -i '/\[Service\]/aEnvironment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock --feature-gates=AllAlpha=true"'  ${unitFile} 
+
+
+systemctl daemon-reload
+systemctl enable kubelet
+
+
+
+exit 0

install/kubernetes/deploy_kubernetes_certificate_approve.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -x
+
+
+export KUBECONFIG=/etc/kubernetes/admin.conf
+
+for i in `seq 1 30`; do
+    aaa=`kubectl get csr --all-namespaces | grep Pending | awk '{print $1}'`
+    if [ "$aaa" ]; then
+        for i in $aaa; do
+            kubectl certificate approve $i || exit 1
+        done
+        sleep 5
+    else
+        break
+    fi
+done
+
+
+exit 0

install/kubernetes/deploy_kubernetes_init_master.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+kubeadm init  --config ${programDir}/kubeadm.yaml 
+
+
+exit 0

install/kubernetes/deploy_kubernetes_install_stackube_addons.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``KUBERNETES_API_PUBLIC_IP`` 
+# - ``CLUSTER_CIDR``, ``CLUSTER_GATEWAY``,
+# - ``KEYSTONE_ADMIN_URL``  must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## install stackube addons
+kubectl -n kube-system delete deployment kube-dns
+kubectl -n kube-system delete daemonset kube-proxy
+
+source /etc/stackube/openstack/admin-openrc.sh
+netList=`openstack network list --long -f value`
+public_network=$(echo "${netList}" | grep External | grep ' public_1 ' | awk '{print $1}')
+[ "${public_network}" ]
+nnn=`echo "${public_network}" | wc -l`
+[ $nnn -eq 1 ]
+
+cinderKeyring=`cat /var/lib/stackube/ceph/ceph_mon_config/ceph.client.cinder.keyring`
+keyring=`echo "${cinderKeyring}" | grep 'key = ' | awk -F\ \=\  '{print $2}'`
+[ "${keyring}" ]
+
+cat > ${programDir}/stackube-configmap.yaml <<EOF
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: stackube-config
+  namespace: kube-system
+data:
+  auth-url: "${KEYSTONE_ADMIN_URL}"
+  username: "admin"
+  password: "${OS_PASSWORD}"
+  tenant-name: "admin"
+  region: "RegionOne"
+  ext-net-id: "${public_network}"
+  plugin-name: "ovs"
+  integration-bridge: "br-int"
+  user-cidr: "${CLUSTER_CIDR}"
+  user-gateway: "${CLUSTER_GATEWAY}"
+  kubernetes-host: "${KUBERNETES_API_PUBLIC_IP}"
+  kubernetes-port: "6443"
+  keyring: "${keyring}"
+EOF
+kubectl create -f ${programDir}/stackube-configmap.yaml 
+kubectl create -f ${programDir}/../../deployment/stackube.yaml
+kubectl create -f ${programDir}/../../deployment/stackube-proxy.yaml
+kubectl create -f ${programDir}/../../deployment/flexvolume/flexvolume-ds.yaml
+
+
+
+
+exit 0

install/kubernetes/kubeadm.yaml

@@ -0,0 +1,19 @@
+kind: MasterConfiguration
+apiVersion: kubeadm.k8s.io/v1alpha1
+kubernetesVersion: "stable"
+controllerManagerExtraArgs:
+  horizontal-pod-autoscaler-use-rest-clients: "true"
+  horizontal-pod-autoscaler-sync-period: "10s"
+  node-monitor-grace-period: "10s"
+  feature-gates: "AllAlpha=true"
+  enable-dynamic-provisioning: "true"
+apiServerExtraArgs:
+  runtime-config: "api/all=true"
+  feature-gates: "AllAlpha=true"
+  experimental-keystone-url: "__KEYSTONE_URL__"
+networking:
+  podSubnet: "__POD_NET_CIDR__"
+api:
+  advertiseAddress: "__KUBERNETES_API_PRIVATE_IP__"
+apiServerCertSANs: ["__KUBERNETES_API_PUBLIC_IP__"]
+

install/kubernetes/remove_kubernetes_from_node.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -x
+
+if command -v kubeadm > /dev/null 2>&1; then
+    kubeadm reset  || exit 1
+fi
+
+
+systemctl stop hyperd kubelet
+yum remove -y  kubelet  kubeadm  kubectl  qemu-hyper  hyperstart  hyper-container  || exit 1
+rm -fr  /etc/kubernetes  /var/lib/kubelet  /var/run/kubernetes
+
+systemctl stop frakti
+rm -f  /usr/bin/frakti  /lib/systemd/system/frakti.service  || exit 1
+systemctl daemon-reload
+
+
+
+exit 0
+

install/lib_common.sh

@@ -0,0 +1,196 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Distro Functions
+# ================
+
+# Determine OS Vendor, Release and Update
+
+#
+# NOTE : For portability, you almost certainly do not want to use
+# these variables directly!  The "is_*" functions defined below this
+# bundle up compatible platforms under larger umbrellas that we have
+# determinted are compatible enough (e.g. is_ubuntu covers Ubuntu &
+# Debian, is_fedora covers RPM-based distros).  Higher-level functions
+# such as "install_package" further abstract things in better ways.
+#
+# ``os_VENDOR`` - vendor name: ``Ubuntu``, ``Fedora``, etc
+# ``os_RELEASE`` - major release: ``16.04`` (Ubuntu), ``23`` (Fedora)
+# ``os_PACKAGE`` - package type: ``deb`` or ``rpm``
+# ``os_CODENAME`` - vendor's codename for release: ``xenial``
+
+declare -g os_VENDOR os_RELEASE os_PACKAGE os_CODENAME
+
+# Make a *best effort* attempt to install lsb_release packages for the
+# user if not available.  Note can't use generic install_package*
+# because they depend on this!
+function _ensure_lsb_release {
+    if [[ -x $(command -v lsb_release 2>/dev/null) ]]; then
+        return
+    fi
+
+    if [[ -x $(command -v apt-get 2>/dev/null) ]]; then
+        sudo apt-get install -y lsb-release
+    elif [[ -x $(command -v zypper 2>/dev/null) ]]; then
+        # XXX: old code paths seem to have assumed SUSE platforms also
+        # had "yum".  Keep this ordered above yum so we don't try to
+        # install the rh package.  suse calls it just "lsb"
+        sudo zypper -n install lsb
+    elif [[ -x $(command -v dnf 2>/dev/null) ]]; then
+        sudo dnf install -y redhat-lsb-core
+    elif [[ -x $(command -v yum 2>/dev/null) ]]; then
+        # all rh patforms (fedora, centos, rhel) have this pkg
+        sudo yum install -y redhat-lsb-core
+    else
+        die $LINENO "Unable to find or auto-install lsb_release"
+    fi
+}
+
+# GetOSVersion
+#  Set the following variables:
+#  - os_RELEASE
+#  - os_CODENAME
+#  - os_VENDOR
+#  - os_PACKAGE
+function GetOSVersion {
+    # We only support distros that provide a sane lsb_release
+    _ensure_lsb_release
+
+    os_RELEASE=$(lsb_release -r -s)
+    os_CODENAME=$(lsb_release -c -s)
+    os_VENDOR=$(lsb_release -i -s)
+
+    if [[ $os_VENDOR =~ (Debian|Ubuntu|LinuxMint) ]]; then
+        os_PACKAGE="deb"
+    else
+        os_PACKAGE="rpm"
+    fi
+
+    typeset -xr os_VENDOR
+    typeset -xr os_RELEASE
+    typeset -xr os_PACKAGE
+    typeset -xr os_CODENAME
+}
+
+# Translate the OS version values into common nomenclature
+# Sets global ``DISTRO`` from the ``os_*`` values
+declare -g DISTRO
+
+function GetDistro {
+    GetOSVersion
+    if [[ "$os_VENDOR" =~ (Ubuntu) || "$os_VENDOR" =~ (Debian) || \
+            "$os_VENDOR" =~ (LinuxMint) ]]; then
+        # 'Everyone' refers to Ubuntu / Debian / Mint releases by
+        # the code name adjective
+        DISTRO=$os_CODENAME
+    elif [[ "$os_VENDOR" =~ (Fedora) ]]; then
+        # For Fedora, just use 'f' and the release
+        DISTRO="f$os_RELEASE"
+    elif [[ "$os_VENDOR" =~ (openSUSE) ]]; then
+        DISTRO="opensuse-$os_RELEASE"
+    elif [[ "$os_VENDOR" =~ (SUSE LINUX) ]]; then
+        # just use major release
+        DISTRO="sle${os_RELEASE%.*}"
+    elif [[ "$os_VENDOR" =~ (Red.*Hat) || \
+        "$os_VENDOR" =~ (CentOS) || \
+        "$os_VENDOR" =~ (Scientific) || \
+        "$os_VENDOR" =~ (OracleServer) || \
+        "$os_VENDOR" =~ (Virtuozzo) ]]; then
+        # Drop the . release as we assume it's compatible
+        # XXX re-evaluate when we get RHEL10
+        DISTRO="rhel${os_RELEASE::1}"
+    elif [[ "$os_VENDOR" =~ (XenServer) ]]; then
+        DISTRO="xs${os_RELEASE%.*}"
+    elif [[ "$os_VENDOR" =~ (kvmibm) ]]; then
+        DISTRO="${os_VENDOR}${os_RELEASE::1}"
+    else
+        # We can't make a good choice here.  Setting a sensible DISTRO
+        # is part of the problem, but not the major issue -- we really
+        # only use DISTRO in the code as a fine-filter.
+        #
+        # The bigger problem is categorising the system into one of
+        # our two big categories as Ubuntu/Debian-ish or
+        # Fedora/CentOS-ish.
+        #
+        # The setting of os_PACKAGE above is only set to "deb" based
+        # on a hard-coded list of vendor names ... thus we will
+        # default to thinking unknown distros are RPM based
+        # (ie. is_ubuntu does not match).  But the platform will then
+        # also not match in is_fedora, because that also has a list of
+        # names.
+        #
+        # So, if you are reading this, getting your distro supported
+        # is really about making sure it matches correctly in these
+        # functions.  Then you can choose a sensible way to construct
+        # DISTRO based on your distros release approach.
+        die $LINENO "Unable to determine DISTRO, can not continue."
+    fi
+    typeset -xr DISTRO
+}
+
+# Utility function for checking machine architecture
+# is_arch arch-type
+function is_arch {
+    [[ "$(uname -m)" == "$1" ]]
+}
+
+# Determine if current distribution is an Oracle distribution
+# is_oraclelinux
+function is_oraclelinux {
+    if [[ -z "$os_VENDOR" ]]; then
+        GetOSVersion
+    fi
+
+    [ "$os_VENDOR" = "OracleServer" ]
+}
+
+
+# Determine if current distribution is a Fedora-based distribution
+# (Fedora, RHEL, CentOS, etc).
+# is_fedora
+function is_fedora {
+    if [[ -z "$os_VENDOR" ]]; then
+        GetOSVersion
+    fi
+
+    [ "$os_VENDOR" = "Fedora" ] || [ "$os_VENDOR" = "Red Hat" ] || \
+        [ "$os_VENDOR" = "RedHatEnterpriseServer" ] || \
+        [ "$os_VENDOR" = "CentOS" ] || [ "$os_VENDOR" = "OracleServer" ] || \
+        [ "$os_VENDOR" = "Virtuozzo" ] || [ "$os_VENDOR" = "kvmibm" ]
+}
+
+
+# Determine if current distribution is a SUSE-based distribution
+# (openSUSE, SLE).
+# is_suse
+function is_suse {
+    if [[ -z "$os_VENDOR" ]]; then
+        GetOSVersion
+    fi
+
+    [[ "$os_VENDOR" =~ (openSUSE) || "$os_VENDOR" == "SUSE LINUX" ]]
+}
+
+
+# Determine if current distribution is an Ubuntu-based distribution
+# It will also detect non-Ubuntu but Debian-based distros
+# is_ubuntu
+function is_ubuntu {
+    if [[ -z "$os_PACKAGE" ]]; then
+        GetOSVersion
+    fi
+    [ "$os_PACKAGE" = "deb" ]
+}

install/lib_tls.sh

@@ -0,0 +1,377 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# !! source _before_ any services that use ``SERVICE_HOST``
+#
+# Dependencies:
+#
+# - ``DEST``, ``DATA_DIR`` must be defined
+# - ``HOST_IP``, ``SERVICE_HOST``
+# - ``KEYSTONE_TOKEN_FORMAT`` must be defined
+
+# Entry points:
+#
+# - configure_CA
+# - init_CA
+# - cleanup_CA
+# - make_root_CA
+# - make_int_CA
+# - make_cert ca-dir cert-name "common-name" ["alt-name" ...]
+
+
+
+# Defaults
+# --------
+
+# TODO: support more distributions
+function is_fedora {
+    # Always true
+    return 0
+}
+
+# Check if this is a valid ipv4 address string
+function is_ipv4_address {
+    local address=$1
+    local regex='([0-9]{1,3}.){3}[0-9]{1,3}'
+    # TODO(clarkb) make this more robust
+    if [[ "$address" =~ $regex ]] ; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+
+SSL_BUNDLE_FILE="$DATA_DIR/ca-bundle.pem"
+TLS_IP=${TLS_IP:-$SERVICE_IP}
+
+STACKUBE_HOSTNAME=$(hostname -f)
+STACKUBE_CERT_NAME=stackube-cert
+STACKUBE_CERT=$DATA_DIR/$STACKUBE_CERT_NAME.pem
+
+# CA configuration
+ROOT_CA_DIR=${ROOT_CA_DIR:-$DATA_DIR/CA/root-ca}
+INT_CA_DIR=${INT_CA_DIR:-$DATA_DIR/CA/int-ca}
+
+ORG_NAME="OpenStack"
+ORG_UNIT_NAME="Stackube"
+
+
+# CA Functions
+# ============
+
+# There may be more than one, get specific
+OPENSSL=${OPENSSL:-/usr/bin/openssl}
+
+# Do primary CA configuration
+function configure_CA {
+    # build common config file
+
+    # Verify ``TLS_IP`` is good
+    if [[ -n "$HOST_IP" && "$HOST_IP" != "$TLS_IP" ]]; then
+        # auto-discover has changed the IP
+        TLS_IP=$HOST_IP
+    fi
+}
+
+# Creates a new CA directory structure
+# create_CA_base ca-dir
+function create_CA_base {
+    local ca_dir=$1
+
+    if [[ -d $ca_dir ]]; then
+        # Bail out it exists
+        return 0
+    fi
+
+    local i
+    for i in certs crl newcerts private; do
+        mkdir -p $ca_dir/$i
+    done
+    chmod 710 $ca_dir/private
+    echo "01" >$ca_dir/serial
+    cp /dev/null $ca_dir/index.txt
+}
+
+# Create a new CA configuration file
+# create_CA_config ca-dir common-name
+function create_CA_config {
+    local ca_dir=$1
+    local common_name=$2
+
+    echo "
+[ ca ]
+default_ca = CA_default
+
+[ CA_default ]
+dir                     = $ca_dir
+policy                  = policy_match
+database                = \$dir/index.txt
+serial                  = \$dir/serial
+certs                   = \$dir/certs
+crl_dir                 = \$dir/crl
+new_certs_dir           = \$dir/newcerts
+certificate             = \$dir/cacert.pem
+private_key             = \$dir/private/cacert.key
+RANDFILE                = \$dir/private/.rand
+default_md              = sha256
+
+[ req ]
+default_bits            = 2048
+default_md              = sha256
+
+prompt                  = no
+distinguished_name      = ca_distinguished_name
+
+x509_extensions         = ca_extensions
+
+[ ca_distinguished_name ]
+organizationName        = $ORG_NAME
+organizationalUnitName  = $ORG_UNIT_NAME Certificate Authority
+commonName              = $common_name
+
+[ policy_match ]
+countryName             = optional
+stateOrProvinceName     = optional
+organizationName        = match
+organizationalUnitName  = optional
+commonName              = supplied
+
+[ ca_extensions ]
+basicConstraints        = critical,CA:true
+subjectKeyIdentifier    = hash
+authorityKeyIdentifier  = keyid:always, issuer
+keyUsage                = cRLSign, keyCertSign
+
+" >$ca_dir/ca.conf
+}
+
+# Create a new signing configuration file
+# create_signing_config ca-dir
+function create_signing_config {
+    local ca_dir=$1
+
+    echo "
+[ ca ]
+default_ca = CA_default
+
+[ CA_default ]
+dir                     = $ca_dir
+policy                  = policy_match
+database                = \$dir/index.txt
+serial                  = \$dir/serial
+certs                   = \$dir/certs
+crl_dir                 = \$dir/crl
+new_certs_dir           = \$dir/newcerts
+certificate             = \$dir/cacert.pem
+private_key             = \$dir/private/cacert.key
+RANDFILE                = \$dir/private/.rand
+default_md              = default
+
+[ req ]
+default_bits            = 1024
+default_md              = sha1
+
+prompt                  = no
+distinguished_name      = req_distinguished_name
+
+x509_extensions         = req_extensions
+
+[ req_distinguished_name ]
+organizationName        = $ORG_NAME
+organizationalUnitName  = $ORG_UNIT_NAME Server Farm
+
+[ policy_match ]
+countryName             = optional
+stateOrProvinceName     = optional
+organizationName        = match
+organizationalUnitName  = optional
+commonName              = supplied
+
+[ req_extensions ]
+basicConstraints        = CA:false
+subjectKeyIdentifier    = hash
+authorityKeyIdentifier  = keyid:always, issuer
+keyUsage                = digitalSignature, keyEncipherment, keyAgreement
+extendedKeyUsage        = serverAuth, clientAuth
+subjectAltName          = \$ENV::SUBJECT_ALT_NAME
+
+" >$ca_dir/signing.conf
+}
+
+# Create root and intermediate CAs
+# init_CA
+function init_CA {
+    # Ensure CAs are built
+    make_root_CA $ROOT_CA_DIR
+    make_int_CA $INT_CA_DIR $ROOT_CA_DIR
+
+    # Create the CA bundle
+    cat $ROOT_CA_DIR/cacert.pem $INT_CA_DIR/cacert.pem >>$INT_CA_DIR/ca-chain.pem
+    cat $INT_CA_DIR/ca-chain.pem >> $SSL_BUNDLE_FILE
+
+    if is_fedora; then
+        sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/ca-trust-source/anchors/stackube-chain.pem
+        sudo update-ca-trust
+    elif is_suse; then
+        sudo cp $INT_CA_DIR/ca-chain.pem /usr/share/pki/trust/anchors/stackube-chain.pem
+        sudo update-ca-certificates
+    elif is_ubuntu; then
+        sudo cp $INT_CA_DIR/ca-chain.pem /usr/local/share/ca-certificates/stackube-int.crt
+        sudo cp $ROOT_CA_DIR/cacert.pem /usr/local/share/ca-certificates/stackube-root.crt
+        sudo update-ca-certificates
+    fi
+}
+
+# Create an initial server cert
+# init_cert
+function init_cert {
+    if [[ ! -r $STACKUBE_CERT ]]; then
+        if [[ -n "$TLS_IP" ]]; then
+            # Lie to let incomplete match routines work
+            TLS_IP="DNS:$TLS_IP,IP:$TLS_IP"
+        fi
+        make_cert $INT_CA_DIR $STACKUBE_CERT_NAME $STACKUBE_HOSTNAME "$TLS_IP"
+
+        # Create a cert bundle
+        cat $INT_CA_DIR/private/$STACKUBE_CERT_NAME.key $INT_CA_DIR/$STACKUBE_CERT_NAME.crt $INT_CA_DIR/cacert.pem >$STACKUBE_CERT
+    fi
+}
+
+# make_cert creates and signs a new certificate with the given commonName and CA
+# make_cert ca-dir cert-name "common-name" ["alt-name" ...]
+function make_cert {
+    local ca_dir=$1
+    local cert_name=$2
+    local common_name=$3
+    local alt_names=$4
+
+    if [ "$common_name" != "$SERVICE_HOST" ]; then
+        if [[ -z "$alt_names" ]]; then
+            alt_names="DNS:$SERVICE_HOST"
+        else
+            alt_names="$alt_names,DNS:$SERVICE_HOST"
+        fi
+        if is_ipv4_address "$SERVICE_HOST" ; then
+            alt_names="$alt_names,IP:$SERVICE_HOST"
+        fi
+    fi
+
+    # Only generate the certificate if it doesn't exist yet on the disk
+    if [ ! -r "$ca_dir/$cert_name.crt" ]; then
+        # Generate a signing request
+        $OPENSSL req \
+            -sha1 \
+            -newkey rsa \
+            -nodes \
+            -keyout $ca_dir/private/$cert_name.key \
+            -out $ca_dir/$cert_name.csr \
+            -subj "/O=${ORG_NAME}/OU=${ORG_UNIT_NAME} Servers/CN=${common_name}"
+
+        if [[ -z "$alt_names" ]]; then
+            alt_names="DNS:${common_name}"
+        else
+            alt_names="DNS:${common_name},${alt_names}"
+        fi
+
+        # Sign the request valid for 1 year
+        SUBJECT_ALT_NAME="$alt_names" \
+        $OPENSSL ca -config $ca_dir/signing.conf \
+            -extensions req_extensions \
+            -days 3650 \
+            -notext \
+            -in $ca_dir/$cert_name.csr \
+            -out $ca_dir/$cert_name.crt \
+            -subj "/O=${ORG_NAME}/OU=${ORG_UNIT_NAME} Servers/CN=${common_name}" \
+            -batch
+    fi
+}
+
+# Make an intermediate CA to sign everything else
+# make_int_CA ca-dir signing-ca-dir
+function make_int_CA {
+    local ca_dir=$1
+    local signing_ca_dir=$2
+
+    # Create the root CA
+    create_CA_base $ca_dir
+    create_CA_config $ca_dir 'Intermediate CA'
+    create_signing_config $ca_dir
+
+    if [ ! -r "$ca_dir/cacert.pem" ]; then
+        # Create a signing certificate request
+        $OPENSSL req -config $ca_dir/ca.conf \
+            -sha1 \
+            -newkey rsa \
+            -nodes \
+            -keyout $ca_dir/private/cacert.key \
+            -out $ca_dir/cacert.csr \
+            -outform PEM
+
+        # Sign the intermediate request valid for 1 year
+        $OPENSSL ca -config $signing_ca_dir/ca.conf \
+            -extensions ca_extensions \
+            -days 3650 \
+            -notext \
+            -in $ca_dir/cacert.csr \
+            -out $ca_dir/cacert.pem \
+            -batch
+    fi
+}
+
+# Make a root CA to sign other CAs
+# make_root_CA ca-dir
+function make_root_CA {
+    local ca_dir=$1
+
+    # Create the root CA
+    create_CA_base $ca_dir
+    create_CA_config $ca_dir 'Root CA'
+
+    if [ ! -r "$ca_dir/cacert.pem" ]; then
+        # Create a self-signed certificate valid for 5 years
+        $OPENSSL req -config $ca_dir/ca.conf \
+            -x509 \
+            -nodes \
+            -newkey rsa \
+            -days 21360 \
+            -keyout $ca_dir/private/cacert.key \
+            -out $ca_dir/cacert.pem \
+            -outform PEM
+    fi
+}
+
+
+
+
+# Cleanup Functions
+# =================
+
+# Clean up the CA files
+# cleanup_CA
+function cleanup_CA {
+    if is_fedora; then
+        sudo rm -f /usr/share/pki/ca-trust-source/anchors/stackube-chain.pem
+        sudo update-ca-trust
+    elif is_ubuntu; then
+        sudo rm -f /usr/local/share/ca-certificates/stackube-int.crt
+        sudo rm -f /usr/local/share/ca-certificates/stackube-root.crt
+        sudo update-ca-certificates
+    fi
+
+    rm -rf "$INT_CA_DIR" "$ROOT_CA_DIR" "$STACKUBE_CERT"
+}
+

install/openstack/config_openstack/cinder-api/cinder.conf

@@ -0,0 +1,56 @@
+[DEFAULT]
+debug = False
+log_dir = /var/log/kolla/cinder
+use_forwarded_for = true
+use_stderr = False
+enable_v1_api = false
+osapi_volume_workers = 4
+volume_name_template = volume-%s
+
+os_region_name = RegionOne
+enabled_backends = rbd-1
+
+osapi_volume_listen = __CINDER_API_IP__
+osapi_volume_listen_port = 8776
+api_paste_config = /etc/cinder/api-paste.ini
+
+auth_strategy = keystone
+transport_url = rabbit://openstack:__RABBITMQ_PWD__@__RABBITMQ_HOST__:5672
+
+[oslo_messaging_notifications]
+driver = noop
+
+[database]
+connection = mysql+pymysql://cinder:__MYSQL_CINDER_PWD__@__MYSQL_HOST__:3306/cinder
+max_retries = -1
+
+[keystone_authtoken]
+auth_uri = https://__OPENSTACK_ENDPOINT_IP__:5001/v3
+auth_url = https://__OPENSTACK_ENDPOINT_IP__:35358/v3
+auth_type = password
+project_domain_id = default
+user_domain_id = default
+project_name = service
+username = cinder
+password = __KEYSTONE_CINDER_PWD__
+cafile = /etc/cinder/haproxy-ca.crt
+
+
+[oslo_concurrency]
+lock_path = /var/lib/cinder/tmp
+
+[rbd-1]
+volume_driver = cinder.volume.drivers.rbd.RBDDriver
+rbd_pool = cinder
+rbd_ceph_conf = /etc/ceph/ceph.conf
+rbd_flatten_volume_from_snapshot = false
+rbd_max_clone_depth = 5
+rbd_store_chunk_size = 4
+rados_connect_timeout = -1
+rbd_user = cinder
+rbd_secret_uuid = b2ec8922-0647-4885-9d6a-94d2688f35a3
+report_discard_supported = True
+
+[privsep_entrypoint]
+helper_command = sudo cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf
+

install/openstack/config_openstack/cinder-api/config.json

@@ -0,0 +1,37 @@
+{
+    "command": "cinder-api --config-file /etc/cinder/cinder.conf",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/cinder.conf",
+            "dest": "/etc/cinder/cinder.conf",
+            "owner": "cinder",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/cinder/policy.json",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/cinder/haproxy-ca.crt",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/lib/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        },
+        {
+            "path": "/var/log/kolla/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/cinder-scheduler/config.json

@@ -0,0 +1,30 @@
+{
+    "command": "cinder-scheduler --config-file /etc/cinder/cinder.conf",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/cinder.conf",
+            "dest": "/etc/cinder/cinder.conf",
+            "owner": "cinder",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/cinder/policy.json",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/lib/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        },
+        {
+            "path": "/var/log/kolla/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/cinder-volume/config.json

@@ -0,0 +1,51 @@
+{
+    "command": "cinder-volume --config-file /etc/cinder/cinder.conf",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/cinder.conf",
+            "dest": "/etc/cinder/cinder.conf",
+            "owner": "cinder",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.*",
+            "dest": "/etc/ceph/",
+            "owner": "cinder",
+            "perm": "0700",
+            "optional": false
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ceph.conf",
+            "dest": "/etc/ceph/ceph.conf",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": false
+        },
+        {
+            "source": "/var/lib/kolla/config_files/nfs_shares",
+            "dest": "/etc/cinder/nfs_shares",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/cinder/policy.json",
+            "owner": "cinder",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/lib/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        },
+        {
+            "path": "/var/log/kolla/cinder",
+            "owner": "cinder:cinder",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/haproxy/config.json

@@ -0,0 +1,18 @@
+{
+    "command": "/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/haproxy.cfg",
+            "dest": "/etc/haproxy/haproxy.cfg",
+            "owner": "root",
+            "perm": "0644"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy.pem",
+            "dest": "/etc/haproxy/haproxy.pem",
+            "owner": "root",
+            "perm": "0600",
+            "optional": false
+        }
+    ]
+}

install/openstack/config_openstack/haproxy/haproxy.cfg

@@ -0,0 +1,61 @@
+global
+  chroot /var/lib/haproxy
+  user haproxy
+  group haproxy
+  daemon
+  log   127.0.0.1 local2
+  maxconn 4000
+  stats socket /var/lib/kolla/haproxy/haproxy.sock
+  ssl-default-bind-ciphers DEFAULT:!MEDIUM:!3DES
+  ssl-default-bind-options no-sslv3 no-tlsv10
+  tune.ssl.default-dh-param 4096
+
+defaults
+  log global
+  mode http
+  option redispatch
+  option httplog
+  option forwardfor
+  retries 3
+  timeout http-request 10s
+  timeout queue 1m
+  timeout connect 10s
+  timeout client 30m
+  timeout server 30m
+  timeout check 10s
+
+
+
+
+listen keystone_internal_external
+  bind __OPENSTACK_ENDPOINT_IP__:5001 ssl crt /etc/haproxy/haproxy.pem
+  http-request del-header X-Forwarded-Proto if { ssl_fc }
+  http-request set-header X-Forwarded-Proto https if { ssl_fc }
+  server stackube __KEYSTONE_API_IP__:5000 check inter 2000 rise 2 fall 5
+
+listen keystone_admin
+  bind __OPENSTACK_ENDPOINT_IP__:35358 ssl crt /etc/haproxy/haproxy.pem
+  http-request del-header X-Forwarded-Proto if { ssl_fc }
+  http-request set-header X-Forwarded-Proto https if { ssl_fc }
+  server stackube __KEYSTONE_API_IP__:35357 check inter 2000 rise 2 fall 5
+
+
+
+listen neutron_server
+  bind __OPENSTACK_ENDPOINT_IP__:9697 ssl crt /etc/haproxy/haproxy.pem
+  server stackube __NEUTRON_API_IP__:9696 check inter 2000 rise 2 fall 5
+
+
+
+listen cinder_api
+  bind __OPENSTACK_ENDPOINT_IP__:8777 ssl crt /etc/haproxy/haproxy.pem
+  http-request del-header X-Forwarded-Proto if { ssl_fc }
+  http-request set-header X-Forwarded-Proto https if { ssl_fc }
+  server stackube __CINDER_API_IP__:8776 check inter 2000 rise 2 fall 5
+
+
+
+
+
+
+

install/openstack/config_openstack/keystone/config.json

@@ -0,0 +1,47 @@
+{
+    "command": "/usr/sbin/httpd",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/keystone.conf",
+            "dest": "/etc/keystone/keystone.conf",
+            "owner": "keystone",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/keystone-paste.ini",
+            "dest": "/etc/keystone/keystone-paste.ini",
+            "owner": "keystone",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/domains",
+            "dest": "/etc/keystone/domains",
+            "owner": "keystone",
+            "perm": "0700",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/keystone/policy.json",
+            "owner": "keystone",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/wsgi-keystone.conf",
+            "dest": "/etc/httpd/conf.d/wsgi-keystone.conf",
+            "owner": "keystone",
+            "perm": "0644"
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla",
+            "owner": "keystone:kolla"
+        },
+        {
+            "path": "/var/log/kolla/keystone/keystone.log",
+            "owner": "keystone:keystone"
+        }
+    ]
+}

install/openstack/config_openstack/keystone/keystone-paste.ini

@@ -0,0 +1,76 @@
+[filter:debug]
+use = egg:oslo.middleware#debug
+
+[filter:request_id]
+use = egg:oslo.middleware#request_id
+
+[filter:build_auth_context]
+use = egg:keystone#build_auth_context
+
+[filter:token_auth]
+use = egg:keystone#token_auth
+
+[filter:json_body]
+use = egg:keystone#json_body
+
+[filter:cors]
+use = egg:oslo.middleware#cors
+oslo_config_project = keystone
+
+[filter:ec2_extension]
+use = egg:keystone#ec2_extension
+
+[filter:ec2_extension_v3]
+use = egg:keystone#ec2_extension_v3
+
+[filter:s3_extension]
+use = egg:keystone#s3_extension
+
+[filter:url_normalize]
+use = egg:keystone#url_normalize
+
+[filter:sizelimit]
+use = egg:oslo.middleware#sizelimit
+
+[app:public_service]
+use = egg:keystone#public_service
+
+[app:service_v3]
+use = egg:keystone#service_v3
+
+[app:admin_service]
+use = egg:keystone#admin_service
+
+[pipeline:public_api]
+pipeline = cors sizelimit url_normalize request_id build_auth_context token_auth json_body ec2_extension public_service
+
+[pipeline:admin_api]
+pipeline = cors sizelimit url_normalize request_id build_auth_context token_auth json_body ec2_extension s3_extension admin_service
+
+[pipeline:api_v3]
+pipeline = cors sizelimit url_normalize request_id build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3
+
+[app:public_version_service]
+use = egg:keystone#public_version_service
+
+[app:admin_version_service]
+use = egg:keystone#admin_version_service
+
+[pipeline:public_version_api]
+pipeline = cors sizelimit url_normalize public_version_service
+
+[pipeline:admin_version_api]
+pipeline = cors sizelimit url_normalize admin_version_service
+
+[composite:main]
+use = egg:Paste#urlmap
+/v2.0 = public_api
+/v3 = api_v3
+/ = public_version_api
+
+[composite:admin]
+use = egg:Paste#urlmap
+/v2.0 = admin_api
+/v3 = api_v3
+/ = admin_version_api
+

install/openstack/config_openstack/keystone/keystone.conf

@@ -0,0 +1,14 @@
+[DEFAULT]
+debug = False
+log_file = /var/log/kolla/keystone/keystone.log
+use_stderr = True
+secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO
+
+[database]
+connection = mysql+pymysql://keystone:__MYSQL_KWYSTONE_PWD__@__MYSQL_HOST__:3306/keystone
+max_retries = -1
+
+[token]
+revoke_by_id = False
+provider = uuid
+

install/openstack/config_openstack/keystone/wsgi-keystone.conf

@@ -0,0 +1,30 @@
+Listen __KEYSTONE_API_IP__:5000
+Listen __KEYSTONE_API_IP__:35357
+
+<VirtualHost *:5000>
+    WSGIDaemonProcess keystone-public processes=4 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
+    WSGIProcessGroup keystone-public
+    WSGIScriptAlias / /var/www/cgi-bin/keystone/main
+    WSGIApplicationGroup %{GLOBAL}
+    WSGIPassAuthorization On
+    <IfVersion >= 2.4>
+      ErrorLogFormat "%{cu}t %M"
+    </IfVersion>
+    ErrorLog "/var/log/kolla/keystone/keystone-apache-public-error.log"
+    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
+    CustomLog "/var/log/kolla/keystone/keystone-apache-public-access.log" logformat
+</VirtualHost>
+
+<VirtualHost *:35357>
+    WSGIDaemonProcess keystone-admin processes=4 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
+    WSGIProcessGroup keystone-admin
+    WSGIScriptAlias / /var/www/cgi-bin/keystone/admin
+    WSGIApplicationGroup %{GLOBAL}
+    WSGIPassAuthorization On
+    <IfVersion >= 2.4>
+      ErrorLogFormat "%{cu}t %M"
+    </IfVersion>
+    ErrorLog "/var/log/kolla/keystone/keystone-apache-admin-error.log"
+    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
+    CustomLog "/var/log/kolla/keystone/keystone-apache-admin-access.log" logformat
+</VirtualHost>

install/openstack/config_openstack/kolla-toolbox/config.json

@@ -0,0 +1,10 @@
+{
+    "command": "sleep infinity",
+    "config_files": [],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/ansible.log",
+            "owner": "ansible:ansible"
+        }
+    ]
+}

install/openstack/config_openstack/neutron-dhcp-agent/config.json

@@ -0,0 +1,55 @@
+{
+    "command": "neutron-dhcp-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/dhcp_agent.ini",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/neutron.conf",
+            "dest": "/etc/neutron/neutron.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ml2_conf.ini",
+            "dest": "/etc/neutron/plugins/ml2/ml2_conf.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/dhcp_agent.ini",
+            "dest": "/etc/neutron/dhcp_agent.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/dnsmasq.conf",
+            "dest": "/etc/neutron/dnsmasq.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/neutron/policy.json",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/neutron/haproxy-ca.crt",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/neutron",
+            "owner": "neutron:neutron",
+            "recurse": true
+        },
+        {
+            "path": "/var/lib/neutron/kolla",
+            "owner": "neutron:neutron",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/neutron-dhcp-agent/dhcp_agent.ini

@@ -0,0 +1,6 @@
+[DEFAULT]
+dnsmasq_config_file = /etc/neutron/dnsmasq.conf
+enable_isolated_metadata = False
+force_metadata = False
+dnsmasq_dns_servers = 8.8.8.8,8.8.4.4
+

install/openstack/config_openstack/neutron-dhcp-agent/dnsmasq.conf

@@ -0,0 +1 @@
+log-facility=/var/log/kolla/neutron/dnsmasq.log

install/openstack/config_openstack/neutron-l3-agent/config.json

@@ -0,0 +1,55 @@
+{
+    "command": "neutron-l3-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l3_agent.ini --config-file /etc/neutron/fwaas_driver.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/neutron.conf",
+            "dest": "/etc/neutron/neutron.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ml2_conf.ini",
+            "dest": "/etc/neutron/plugins/ml2/ml2_conf.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/fwaas_driver.ini",
+            "dest": "/etc/neutron/fwaas_driver.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/l3_agent.ini",
+            "dest": "/etc/neutron/l3_agent.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/neutron/policy.json",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/neutron/haproxy-ca.crt",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/neutron",
+            "owner": "neutron:neutron",
+            "recurse": true
+        },
+        {
+            "path": "/var/lib/neutron/kolla",
+            "owner": "neutron:neutron",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/neutron-l3-agent/fwaas_driver.ini

@@ -0,0 +1,2 @@
+[fwaas]
+

install/openstack/config_openstack/neutron-l3-agent/l3_agent.ini

@@ -0,0 +1,3 @@
+[DEFAULT]
+agent_mode = legacy
+

install/openstack/config_openstack/neutron-lbaas-agent/config.json

@@ -0,0 +1,55 @@
+{
+    "command": "neutron-lbaasv2-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/lbaas_agent.ini --config-file /etc/neutron/neutron_lbaas.conf",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/neutron.conf",
+            "dest": "/etc/neutron/neutron.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/lbaas_agent.ini",
+            "dest": "/etc/neutron/lbaas_agent.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ml2_conf.ini",
+            "dest": "/etc/neutron/plugins/ml2/ml2_conf.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/neutron_lbaas.conf",
+            "dest": "/etc/neutron/neutron_lbaas.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/neutron/policy.json",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/neutron/haproxy-ca.crt",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/neutron",
+            "owner": "neutron:neutron",
+            "recurse": true
+        },
+        {
+            "path": "/var/lib/neutron/kolla",
+            "owner": "neutron:neutron",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/neutron-lbaas-agent/lbaas_agent.ini

@@ -0,0 +1,7 @@
+[DEFAULT]
+debug = False
+device_driver = neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver
+
+[haproxy]
+user_group = haproxy
+

install/openstack/config_openstack/neutron-openvswitch-agent/config.json

@@ -0,0 +1,38 @@
+{
+    "command": "neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/neutron.conf",
+            "dest": "/etc/neutron/neutron.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ml2_conf.ini",
+            "dest": "/etc/neutron/plugins/ml2/ml2_conf.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/neutron/policy.json",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/neutron/haproxy-ca.crt",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/neutron",
+            "owner": "neutron:neutron",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/neutron-server/config.json

@@ -0,0 +1,50 @@
+{
+    "command": "neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/neutron_vpnaas.conf",
+    "config_files": [
+        {
+            "source": "/var/lib/kolla/config_files/neutron.conf",
+            "dest": "/etc/neutron/neutron.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/neutron_lbaas.conf",
+            "dest": "/etc/neutron/neutron_lbaas.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/neutron_vpnaas.conf",
+            "dest": "/etc/neutron/neutron_vpnaas.conf",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/ml2_conf.ini",
+            "dest": "/etc/neutron/plugins/ml2/ml2_conf.ini",
+            "owner": "neutron",
+            "perm": "0600"
+        },
+        {
+            "source": "/var/lib/kolla/config_files/policy.json",
+            "dest": "/etc/neutron/policy.json",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        },
+        {
+            "source": "/var/lib/kolla/config_files/haproxy-ca.crt",
+            "dest": "/etc/neutron/haproxy-ca.crt",
+            "owner": "neutron",
+            "perm": "0600",
+            "optional": true
+        }
+    ],
+    "permissions": [
+        {
+            "path": "/var/log/kolla/neutron",
+            "owner": "neutron:neutron",
+            "recurse": true
+        }
+    ]
+}

install/openstack/config_openstack/neutron-server/ml2_conf.ini

@@ -0,0 +1,28 @@
+[ml2]
+type_drivers = flat,vlan,vxlan
+tenant_network_types = vxlan
+mechanism_drivers = openvswitch,l2population
+
+[ml2_type_vlan]
+network_vlan_ranges =
+
+[ml2_type_flat]
+flat_networks = physnet1
+
+[ml2_type_vxlan]
+vni_ranges = 1:10000
+vxlan_group = 239.1.1.1
+
+[securitygroup]
+firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
+
+[agent]
+tunnel_types = vxlan
+l2_population = true
+arp_responder = true
+
+[ovs]
+bridge_mappings = physnet1:br-ex
+ovsdb_connection = tcp:__OVSDB_IP__:6640
+local_ip = __LOCAL_IP__
+

install/openstack/config_openstack/neutron-server/neutron.conf

@@ -0,0 +1,41 @@
+[DEFAULT]
+debug = False
+log_dir = /var/log/kolla/neutron
+use_stderr = False
+bind_host = __NEUTRON_API_IP__
+bind_port = 9696
+api_paste_config = /usr/share/neutron/api-paste.ini
+endpoint_type = internalURL
+api_workers = 4
+interface_driver = openvswitch
+allow_overlapping_ips = true
+core_plugin = ml2
+service_plugins = router,neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2
+transport_url = rabbit://openstack:__RABBITMQ_PWD__@__RABBITMQ_HOST__:5672
+
+
+[oslo_concurrency]
+lock_path = /var/lib/neutron/tmp
+
+[agent]
+root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
+
+[database]
+connection = mysql+pymysql://neutron:__MYSQL_NEUTRON_PWD__@__MYSQL_HOST__:3306/neutron
+max_retries = -1
+
+[keystone_authtoken]
+auth_uri = https://__OPENSTACK_ENDPOINT_IP__:5001/v3
+auth_url = https://__OPENSTACK_ENDPOINT_IP__:35358/v3
+auth_type = password
+project_domain_id = default
+user_domain_id = default
+project_name = service
+username = neutron
+password = __KEYSTONE_NEUTRON_PWD__
+cafile = /etc/neutron/haproxy-ca.crt
+
+
+[oslo_messaging_notifications]
+driver = noop
+

install/openstack/config_openstack/neutron-server/neutron_lbaas.conf

@@ -0,0 +1,12 @@
+[service_providers]
+service_provider = LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
+
+[service_auth]
+auth_url = http://__KEYSTONE_API_IP__:5000/v2.0
+admin_tenant_name = service
+admin_user = neutron
+admin_password = __NEUTRON_KEYSTONE_PWD__
+auth_version = 2
+region = RegionOne
+endpoint_type = internal
+

install/openstack/config_openstack/openvswitch-db-server/config.json

@@ -0,0 +1,4 @@
+{
+    "command": "start-ovsdb-server __OVSDB_IP__ ",
+    "config_files": []
+}

install/openstack/config_openstack/openvswitch-vswitchd/config.json

@@ -0,0 +1,4 @@
+{
+    "command": "/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log",
+    "config_files": []
+}

install/openstack/deploy_openstack_cinder_api.sh

@@ -0,0 +1,159 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OPENSTACK_ENDPOINT_IP``
+# - ``MYSQL_HOST``, ``MYSQL_ROOT_PWD``
+# - ``KEYSTONE_ADMIN_PWD``
+# - ``KEYSTONE_CINDER_PWD``, ``MYSQL_CINDER_PWD``must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## register - Creating the Cinder service and endpoint
+## v1
+for IF in 'admin' 'internal' 'public'; do
+    echo ${IF}
+    docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost  -m kolla_keystone_service \
+        -a "service_name=cinder
+            service_type=volume
+            description='Openstack Block Storage'
+            endpoint_region=RegionOne
+            url='https://${OPENSTACK_ENDPOINT_IP}:8777/v1/%(tenant_id)s'
+            interface='${IF}'
+            region_name=RegionOne
+            auth='{{ openstack_keystone_auth }}'
+            verify=False  " \
+        -e "{'openstack_keystone_auth': {
+               'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+               'username': 'admin',
+               'password': '${KEYSTONE_ADMIN_PWD}',
+               'project_name': 'admin',
+               'domain_name': 'default' } 
+            }"
+done
+
+## v2
+for VER in 'v2' ; do
+    echo -e "\n--- ${VER} ---"
+    for IF in 'admin' 'internal' 'public'; do
+        echo ${IF}
+        docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost  -m kolla_keystone_service \
+            -a "service_name=cinder${VER}
+                service_type=volume${VER}
+                description='Openstack Block Storage'
+                endpoint_region=RegionOne
+                url='https://${OPENSTACK_ENDPOINT_IP}:8777/${VER}/%(tenant_id)s'
+                interface='${IF}'
+                region_name=RegionOne
+                auth='{{ openstack_keystone_auth }}'
+                verify=False  " \
+            -e "{'openstack_keystone_auth': {
+                   'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+                   'username': 'admin',
+                   'password': '${KEYSTONE_ADMIN_PWD}',
+                   'project_name': 'admin',
+                   'domain_name': 'default' } 
+                }"
+    done
+done
+
+
+## register -  Creating the Cinder project, user, and role
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost  -m kolla_keystone_user \
+    -a "project=service
+        user=cinder
+        password=${KEYSTONE_CINDER_PWD}
+        role=admin
+        region_name=RegionOne
+        auth='{{ openstack_keystone_auth }}'
+        verify=False  " \
+    -e "{'openstack_keystone_auth': {
+           'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+           'username': 'admin',
+           'password': '${KEYSTONE_ADMIN_PWD}',
+           'project_name': 'admin',
+           'domain_name': 'default' } 
+        }"
+
+
+
+# bootstrap - Creating Cinder database
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost   -m mysql_db \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=cinder"
+
+# bootstrap - Creating Cinder database user and setting permissions
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost   -m mysql_user \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=cinder
+        password=${MYSQL_CINDER_PWD}
+        host=%
+        priv='cinder.*:ALL'
+        append_privs=yes"
+
+
+
+# bootstrap_service - Running Cinder bootstrap container
+docker run --net host  \
+    --name stackube_openstack_bootstrap_cinder  \
+    -v /etc/stackube/openstack/cinder-api/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -e "KOLLA_BOOTSTRAP="  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    kolla/centos-binary-cinder-api:4.0.0
+
+sleep 2
+docker rm stackube_openstack_bootstrap_cinder
+
+
+## start_container - cinder-api
+docker run -d  --net host  \
+    --name stackube_openstack_cinder_api  \
+    -v /etc/stackube/openstack/cinder-api/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=cinder-api"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-cinder-api:4.0.0
+
+sleep 5
+
+
+exit 0

install/openstack/deploy_openstack_cinder_scheduler.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``API_IP``, ``RABBITMQ_PWD``
+# - ``KEYSTONE_ADMIN_PWD``
+# - ``KEYSTONE_CINDER_PWD``, ``MYSQL_CINDER_PWD``must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## start_container - cinder-scheduler
+docker run -d  --net host  \
+    --name stackube_openstack_cinder_scheduler  \
+    -v /etc/stackube/openstack/cinder-scheduler/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=cinder-scheduler"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-cinder-scheduler:4.0.0
+
+sleep 5
+
+
+
+exit 0

install/openstack/deploy_openstack_cinder_volume.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``API_IP``, ``RABBITMQ_PWD``
+# - ``KEYSTONE_ADMIN_PWD``
+# - ``KEYSTONE_CINDER_PWD``, ``MYSQL_CINDER_PWD``must be defined
+#
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## start_container - cinder-volume
+docker run -d  --net host  \
+    --name stackube_openstack_cinder_volume  \
+    -v /etc/stackube/openstack/cinder-volume/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run/:/run/:shared  \
+    -v /dev/:/dev/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=cinder-volume"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-cinder-volume:4.0.0
+
+sleep 5
+
+
+
+exit 0

install/openstack/deploy_openstack_haproxy.sh

@@ -0,0 +1,78 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OPENSTACK_ENDPOINT_IP``
+# - ``KEYSTONE_API_IP``
+# - ``NEUTRON_API_IP``
+# - ``CINDER_API_IP``  must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## make certificates
+HOST_IP=${OPENSTACK_ENDPOINT_IP}
+SERVICE_HOST=${OPENSTACK_ENDPOINT_IP}
+SERVICE_IP=${OPENSTACK_ENDPOINT_IP}
+DATA_DIR='/etc/stackube/openstack/certificates'
+source ${programDir}/../lib_tls.sh
+mkdir -p ${DATA_DIR}
+init_CA
+init_cert
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## config files
+mkdir -p /etc/stackube/openstack
+cp -a ${programDir}/config_openstack/haproxy /etc/stackube/openstack/
+sed -i "s/__OPENSTACK_ENDPOINT_IP__/${OPENSTACK_ENDPOINT_IP}/g" /etc/stackube/openstack/haproxy/haproxy.cfg
+sed -i "s/__KEYSTONE_API_IP__/${KEYSTONE_API_IP}/g" /etc/stackube/openstack/haproxy/haproxy.cfg
+sed -i "s/__NEUTRON_API_IP__/${NEUTRON_API_IP}/g" /etc/stackube/openstack/haproxy/haproxy.cfg
+sed -i "s/__CINDER_API_IP__/${CINDER_API_IP}/g" /etc/stackube/openstack/haproxy/haproxy.cfg
+# STACKUBE_CERT defined in lib_tls.sh
+cat ${STACKUBE_CERT} > /etc/stackube/openstack/haproxy/haproxy.pem
+
+
+## run
+docker run -d  --net host  \
+    --name stackube_openstack_haproxy  \
+    -v /etc/stackube/openstack/haproxy/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=haproxy"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-haproxy:4.0.0
+
+
+exit 0
+

install/openstack/deploy_openstack_keystone.sh

@@ -0,0 +1,121 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OPENSTACK_ENDPOINT_IP``, ``KEYSTONE_API_IP``
+# - ``MYSQL_HOST``, ``MYSQL_ROOT_PWD``
+# - ``MYSQL_KEYSTONE_PWD``, ``KEYSTONE_ADMIN_PWD``  must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## create db
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost -m mysql_db  \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=keystone"
+
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost -m mysql_user  \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=keystone
+        password=${MYSQL_KEYSTONE_PWD}
+        host=%
+        priv=keystone.*:ALL
+        append_privs=yes "
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## config files
+mkdir -p /etc/stackube/openstack
+cp -a ${programDir}/config_openstack/keystone /etc/stackube/openstack/
+sed -i "s/__MYSQL_HOST__/${MYSQL_HOST}/g" /etc/stackube/openstack/keystone/keystone.conf
+sed -i "s/__MYSQL_KWYSTONE_PWD__/${MYSQL_KEYSTONE_PWD}/g" /etc/stackube/openstack/keystone/keystone.conf
+sed -i "s/__KEYSTONE_API_IP__/${KEYSTONE_API_IP}/g" /etc/stackube/openstack/keystone/wsgi-keystone.conf
+
+
+# bootstrap_service
+docker run --net host  \
+    --name stackube_openstack_bootstrap_keystone  \
+    -v /etc/stackube/openstack/keystone/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -e "KOLLA_BOOTSTRAP="  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    kolla/centos-binary-keystone:4.0.0
+
+docker rm stackube_openstack_bootstrap_keystone
+
+docker run -d  --net host  \
+    --name stackube_openstack_keystone  \
+    -v /etc/stackube/openstack/keystone/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -e "KOLLA_SERVICE_NAME=keystone"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    --restart unless-stopped \
+    kolla/centos-binary-keystone:4.0.0
+
+sleep 10
+
+# register
+docker exec stackube_openstack_keystone kolla_keystone_bootstrap admin ${KEYSTONE_ADMIN_PWD} admin admin \
+    https://${OPENSTACK_ENDPOINT_IP}:35358/v3 \
+    https://${OPENSTACK_ENDPOINT_IP}:5001/v3 \
+    https://${OPENSTACK_ENDPOINT_IP}:5001/v3 \
+    RegionOne
+
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost -m os_keystone_role  -a "name=_member_  auth='{{ openstack_keystone_auth }}' verify=False"  \
+    -e "{'openstack_keystone_auth': {
+           'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+           'username': 'admin',
+           'password': '${KEYSTONE_ADMIN_PWD}',
+           'project_name': 'admin',
+           'domain_name': 'default' } 
+        }"
+
+
+cat > /etc/stackube/openstack/admin-openrc.sh << EOF
+export OS_PROJECT_DOMAIN_NAME=default
+export OS_USER_DOMAIN_NAME=default
+export OS_PROJECT_NAME=admin
+export OS_TENANT_NAME=admin
+export OS_USERNAME=admin
+export OS_PASSWORD=${KEYSTONE_ADMIN_PWD}
+export OS_AUTH_URL=https://${OPENSTACK_ENDPOINT_IP}:35358/v3
+export OS_INTERFACE=internal
+export OS_IDENTITY_API_VERSION=3
+export OS_CACERT=/etc/stackube/openstack/certificates/CA/int-ca/ca-chain.pem
+EOF
+
+exit 0
+

install/openstack/deploy_openstack_kolla_toolbox.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## kolla-toolbox
+docker run -d  --net host  \
+    --name stackube_openstack_kolla_toolbox  \
+    -v /run/:/run/:shared  \
+    -v /dev/:/dev/:rw  \
+    -v /etc/stackube/openstack/kolla-toolbox/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -e "KOLLA_SERVICE_NAME=kolla-toolbox"  \
+    -e "ANSIBLE_LIBRARY=/usr/share/ansible"  \
+    -e "ANSIBLE_NOCOLOR=1"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    --restart unless-stopped  \
+    --privileged  \
+    kolla/centos-binary-kolla-toolbox:4.0.0
+
+sleep 5
+
+
+exit 0
+

install/openstack/deploy_openstack_mariadb.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``MYSQL_ROOT_PWD`` must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## mariadb
+mkdir -p /var/lib/stackube/openstack/mariadb  && \
+docker run -d \
+    --name stackube_openstack_mariadb \
+    --net host  \
+    -e MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PWD} \
+    -v /var/lib/stackube/openstack/mariadb:/var/lib/mysql \
+    --restart unless-stopped \
+    mariadb:5.5
+
+sleep 5
+
+exit 0
+

install/openstack/deploy_openstack_neutron_dhcp_agent.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OVSDB_IP``
+# - ``ML2_LOCAL_IP`` must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## start_container - neutron-dhcp-agent
+sed -i "s/__OVSDB_IP__/${OVSDB_IP}/g" /etc/stackube/openstack/neutron-dhcp-agent/ml2_conf.ini
+sed -i "s/__LOCAL_IP__/${ML2_LOCAL_IP}/g" /etc/stackube/openstack/neutron-dhcp-agent/ml2_conf.ini
+
+docker run -d  --net host  \
+    --name stackube_openstack_neutron_dhcp_agent  \
+    -v /etc/stackube/openstack/neutron-dhcp-agent/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run:/run:shared  \
+    \
+    -e "KOLLA_SERVICE_NAME=neutron-dhcp-agent"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-neutron-dhcp-agent:4.0.0
+
+
+
+exit 0

install/openstack/deploy_openstack_neutron_l3_agent.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OVSDB_IP``
+# - ``ML2_LOCAL_IP`` must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## sysctl
+sed -i '/^net\.ipv4\.ip_forward=/d' /etc/sysctl.conf
+sed -i '/^net\.ipv4\.conf\.all\.rp_filter=/d' /etc/sysctl.conf
+sed -i '/^net\.ipv4\.conf\.default\.rp_filter=/d' /etc/sysctl.conf
+
+echo '
+net.ipv4.ip_forward=1
+net.ipv4.conf.all.rp_filter=0
+net.ipv4.conf.default.rp_filter=0
+' >> /etc/sysctl.conf
+
+sysctl -p
+
+
+## start_container - neutron-l3-agent
+sed -i "s/__OVSDB_IP__/${OVSDB_IP}/g" /etc/stackube/openstack/neutron-l3-agent/ml2_conf.ini
+sed -i "s/__LOCAL_IP__/${ML2_LOCAL_IP}/g" /etc/stackube/openstack/neutron-l3-agent/ml2_conf.ini
+
+docker run -d  --net host  \
+    --name stackube_openstack_neutron_l3_agent  \
+    -v /etc/stackube/openstack/neutron-l3-agent/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run:/run:shared  \
+    \
+    -e "KOLLA_SERVICE_NAME=neutron-l3-agent"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-neutron-l3-agent:4.0.0
+
+
+exit 0

install/openstack/deploy_openstack_neutron_lbaas_agent.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OVSDB_IP``, ``ML2_LOCAL_IP``
+# - ``KEYSTONE_API_IP``, ``KEYSTONE_NEUTRON_PWD`` must be defined
+#
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+# bootstrap_service - Running Neutron lbaas bootstrap container
+sed -i "s/__OVSDB_IP__/${OVSDB_IP}/g" /etc/stackube/openstack/neutron-lbaas-agent/ml2_conf.ini
+sed -i "s/__LOCAL_IP__/${ML2_LOCAL_IP}/g" /etc/stackube/openstack/neutron-lbaas-agent/ml2_conf.ini
+
+sed -i "s/__KEYSTONE_API_IP__/${KEYSTONE_API_IP}/g" /etc/stackube/openstack/neutron-lbaas-agent/neutron_lbaas.conf
+sed -i "s/__NEUTRON_KEYSTONE_PWD__/${KEYSTONE_NEUTRON_PWD}/g" /etc/stackube/openstack/neutron-lbaas-agent/neutron_lbaas.conf
+
+docker run --net host  \
+    --name stackube_openstack_bootstrap_neutron_lbaas_agent  \
+    -v /etc/stackube/openstack/neutron-lbaas-agent/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run/netns/:/run/netns/:shared  \
+    -v /run:/run:shared  \
+    \
+    -e "KOLLA_BOOTSTRAP="  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --privileged  \
+    kolla/centos-binary-neutron-lbaas-agent:4.0.0
+
+sleep 2
+docker rm stackube_openstack_bootstrap_neutron_lbaas_agent
+
+
+## start_container - neutron-lbaas-agent
+docker run -d  --net host  \
+    --name stackube_openstack_neutron_lbaas_agent  \
+    -v /etc/stackube/openstack/neutron-lbaas-agent/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run/netns/:/run/netns/:shared  \
+    -v /run:/run:shared  \
+    \
+    -e "KOLLA_SERVICE_NAME=neutron-lbaas-agent"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-neutron-lbaas-agent:4.0.0
+
+
+exit 0

install/openstack/deploy_openstack_neutron_openvswitch_agent.sh

@@ -0,0 +1,98 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OVSDB_IP``
+# - ``ML2_LOCAL_IP`` must be defined
+#
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+## openvswitch-db-server
+sed -i "s/__OVSDB_IP__/${OVSDB_IP}/g" /etc/stackube/openstack/openvswitch-db-server/config.json
+mkdir -p /var/lib/stackube/openstack/openvswitch
+docker run -d  --net host  \
+    --name stackube_openstack_openvswitch_db  \
+    -v /etc/stackube/openstack/openvswitch-db-server/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /var/lib/stackube/openstack/openvswitch/:/var/lib/openvswitch/:rw  \
+    -v /run:/run:shared  \
+    \
+    -e "KOLLA_SERVICE_NAME=openvswitch-db"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-openvswitch-db-server:4.0.0
+
+sleep 5
+
+# config br
+docker exec stackube_openstack_openvswitch_db /usr/local/bin/kolla_ensure_openvswitch_configured br-ex
+
+
+## openvswitch-vswitchd
+docker run -d  --net host  \
+    --name stackube_openstack_openvswitch_vswitchd  \
+    -v /etc/stackube/openstack/openvswitch-vswitchd/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run:/run:shared  \
+    -v /lib/modules:/lib/modules:ro  \
+    \
+    -e "KOLLA_SERVICE_NAME=openvswitch-vswitchd"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-openvswitch-vswitchd:4.0.0
+
+sleep 5
+
+
+## start_container - neutron-openvswitch-agent
+sed -i "s/__OVSDB_IP__/${OVSDB_IP}/g" /etc/stackube/openstack/neutron-openvswitch-agent/ml2_conf.ini
+sed -i "s/__LOCAL_IP__/${ML2_LOCAL_IP}/g" /etc/stackube/openstack/neutron-openvswitch-agent/ml2_conf.ini
+
+
+docker run -d  --net host  \
+    --name stackube_openstack_neutron_openvswitch_agent  \
+    -v /etc/stackube/openstack/neutron-openvswitch-agent/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -v /run:/run:shared  \
+    -v /lib/modules:/lib/modules:ro  \
+    \
+    -e "KOLLA_SERVICE_NAME=neutron-openvswitch-agent"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    --privileged  \
+    kolla/centos-binary-neutron-openvswitch-agent:4.0.0  || exit 1
+
+exit 0

install/openstack/deploy_openstack_neutron_server.sh

@@ -0,0 +1,133 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``OPENSTACK_ENDPOINT_IP``
+# - ``MYSQL_HOST``, ``MYSQL_ROOT_PWD``
+# - ``KEYSTONE_ADMIN_PWD``
+# - ``KEYSTONE_NEUTRON_PWD``, ``MYSQL_NEUTRON_PWD`` must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+
+## register - Creating the Neutron service and endpoint
+for IF in 'admin' 'internal' 'public'; do 
+    docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost  -m kolla_keystone_service \
+        -a "service_name=neutron
+            service_type=network
+            description='Openstack Networking'
+            endpoint_region=RegionOne
+            url='https://${OPENSTACK_ENDPOINT_IP}:9697/'
+            interface='${IF}'
+            region_name=RegionOne
+            auth='{{ openstack_keystone_auth }}'
+            verify=False  " \
+        -e "{'openstack_keystone_auth': {
+               'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+               'username': 'admin',
+               'password': '${KEYSTONE_ADMIN_PWD}',
+               'project_name': 'admin',
+               'domain_name': 'default' } 
+            }" 
+done
+
+
+## register - Creating the Neutron project, user, and role
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost  -m kolla_keystone_user \
+    -a "project=service
+        user=neutron
+        password=${KEYSTONE_NEUTRON_PWD}
+        role=admin
+        region_name=RegionOne
+        auth='{{ openstack_keystone_auth }}'
+        verify=False  " \
+    -e "{'openstack_keystone_auth': {
+           'auth_url': 'https://${OPENSTACK_ENDPOINT_IP}:35358/v3',
+           'username': 'admin',
+           'password': '${KEYSTONE_ADMIN_PWD}',
+           'project_name': 'admin',
+           'domain_name': 'default' } 
+        }" 
+
+
+# bootstrap - Creating Neutron database
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost   -m mysql_db \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=neutron"
+
+# bootstrap - Creating Neutron database user and setting permissions
+docker exec stackube_openstack_kolla_toolbox /usr/bin/ansible localhost   -m mysql_user \
+    -a "login_host=${MYSQL_HOST}
+        login_port=3306
+        login_user=root
+        login_password=${MYSQL_ROOT_PWD}
+        name=neutron
+        password=${MYSQL_NEUTRON_PWD}
+        host=%
+        priv='neutron.*:ALL'
+        append_privs=yes"
+
+
+
+
+## log dir
+mkdir -p /var/log/stackube/openstack
+chmod 777 /var/log/stackube/openstack
+
+
+# bootstrap_service - Running Neutron bootstrap container
+docker run --net host  \
+    --name stackube_openstack_bootstrap_neutron  \
+    -v /etc/stackube/openstack/neutron-server/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    -e "KOLLA_BOOTSTRAP="  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    kolla/centos-binary-neutron-server:4.0.0
+
+sleep 2
+docker rm stackube_openstack_bootstrap_neutron
+
+
+## start_container - neutron-server
+docker run -d  --net host  \
+    --name stackube_openstack_neutron_server  \
+    -v /etc/stackube/openstack/neutron-server/:/var/lib/kolla/config_files/:ro  \
+    -v /var/log/stackube/openstack:/var/log/kolla/:rw  \
+    \
+    -e "KOLLA_SERVICE_NAME=neutron-server"  \
+    -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" \
+    \
+    --restart unless-stopped \
+    kolla/centos-binary-neutron-server:4.0.0
+
+
+
+
+exit 0

install/openstack/deploy_openstack_rabbitmq.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Dependencies:
+#
+# - ``RABBITMQ_PWD`` must be defined
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o nounset
+set -o pipefail
+set -x
+
+## rabbitmq 
+mkdir -p /var/lib/stackube/openstack/rabbitmq  && \
+docker run -d \
+    --name stackube_openstack_rabbitmq \
+    --net host  \
+    -v /var/lib/stackube/openstack/rabbitmq:/var/lib/rabbitmq \
+    --restart unless-stopped \
+    rabbitmq:3.6 || exit 1
+
+sleep 5
+for i in 1 2 3 4 5; do
+    docker exec stackube_openstack_rabbitmq rabbitmqctl status && break
+    sleep $i
+done
+sleep 5
+
+docker exec stackube_openstack_rabbitmq rabbitmqctl add_user openstack ${RABBITMQ_PWD} || exit 1
+docker exec stackube_openstack_rabbitmq rabbitmqctl set_permissions openstack ".*" ".*" ".*" || exit 1
+
+exit 0

install/openstack/remove_openstack_from_node.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -x
+
+## clean certificates
+source ${programDir}/lib_tls.sh || exit 1
+cleanup_CA || exit 1
+
+
+## remove docker containers
+stackubeConstaners=`docker ps -a | awk '{print $NF}' | grep '^stackube_openstack_' `
+if [ "${stackubeConstaners}" ]; then
+    docker rm -f $stackubeConstaners || exit 1
+fi
+
+## rm dirs
+rm -fr /etc/stackube/openstack  /var/log/stackube/openstack  /var/lib/stackube/openstack || exit 1
+
+
+
+exit 0
+

install/remove.sh

@@ -0,0 +1,108 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+
+function usage {
+    echo "
+Usage:
+   bash $(basename $0) CONFIG_FILE
+"
+}
+
+[ "$1" ] || { usage; exit 1; }
+[ -f "$1" ] || { echo "Error: $1 not exists or not a file!"; exit 1; }
+
+source $(readlink -f $1) || { echo "'source $(readlink -f $1)' failed!"; exit 1; }
+
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ] || { echo "Error: CONTROL_NODE_PRIVATE_IP not defined!"; exit 1; }
+[ "${NETWORK_NODES_PRIVATE_IP}" ] || { echo "Error: NETWORK_NODES_PRIVATE_IP not defined!"; exit 1; }
+[ "${COMPUTE_NODES_PRIVATE_IP}" ] || { echo "Error: COMPUTE_NODES_PRIVATE_IP not defined!"; exit 1; }
+[ "${STORAGE_NODES_PRIVATE_IP}" ] || { echo "Error: STORAGE_NODES_PRIVATE_IP not defined!"; exit 1; }
+[ "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" ] || { echo "Error: STORAGE_NODES_CEPH_OSD_DATA_DIR not defined!"; exit 1; }
+
+
+#####################
+
+set -x
+
+
+## log
+logDir='/var/log/stackube'
+logFile="${logDir}/remove.log-$(date '+%Y-%m-%d_%H-%M-%S')"
+mkdir -p ${logDir}
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${NETWORK_NODES_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}
+${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+{
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') remove_kubernetes"
+    remove_kubernetes=''
+    for i in `seq 1 10`; do
+        bash ${programDir}/remove_kubernetes.sh $(readlink -f $1)
+        if [ "$?" == "0" ]; then
+            remove_kubernetes='done'
+            break
+        fi
+    done
+    [ "${remove_kubernetes}" == "done" ] || { echo "Error: remove_kubernetes failed !"; exit 1;  }
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') remove_openstack"
+    remove_openstack=''
+    for i in `seq 1 10`; do
+        bash ${programDir}/remove_openstack.sh $(readlink -f $1)
+        if [ "$?" == "0" ]; then
+            remove_openstack='done'
+            break
+        fi
+    done
+    [ "${remove_openstack}" == "done" ] || { echo "Error: remove_openstack failed !"; exit 1;  }
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') remove_ceph"
+    remove_ceph=''
+    for i in `seq 1 10`; do
+        bash ${programDir}/remove_ceph.sh $(readlink -f $1)
+        if [ "$?" == "0" ]; then
+            remove_ceph='done'
+            break
+        fi
+    done
+    [ "${remove_ceph}" == "done" ] || { echo "Error: remove_ceph failed !"; exit 1;  }
+
+    echo -e "\n$(date '+%Y-%m-%d %H:%M:%S') All done!"
+
+} 2>&1 | tee -a ${logFile}
+
+
+allStats=(${PIPESTATUS[@]})
+if [ "${allStats[0]}" != "0" ]; then
+    exit 1
+fi
+
+
+exit 0
+
+
+
+

install/remove_ceph.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" ]
+
+
+# ceph-mon
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/ceph/remove_ceph_from_node.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/remove_ceph_from_node.sh"
+done
+
+
+
+# ceph-osd
+storageIpList=(`echo "${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g'`)
+osdDataDirList=(`echo "${STORAGE_NODES_CEPH_OSD_DATA_DIR}" | sed -e 's/,/\n/g'`)
+[ ${#storageIpList[@]} -eq ${#osdDataDirList[@]} ]
+
+MAX=$((${#storageIpList[@]} - 1))
+for i in `seq 0 ${MAX}`; do
+    IP="${storageIpList[$i]}"
+    dataDir="${osdDataDirList[$i]}"
+    echo -e "\n------ ${IP} ${dataDir} ------"
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/ceph/remove_ceph_from_node.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "export CEPH_OSD_DATA_DIR='${dataDir}'
+                    /bin/bash /tmp/stackube_install/remove_ceph_from_node.sh"
+done
+
+
+
+exit 0
+

install/remove_kubernetes.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+[ "${COMPUTE_NODES_PRIVATE_IP}" ]
+
+
+
+## all nodes
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+# hyperd frakti
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/kubernetes/remove_kubernetes_from_node.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/remove_kubernetes_from_node.sh"
+done
+
+
+exit 0
+

install/remove_openstack.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+# Copyright (c) 2017 OpenStack Foundation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+programDir=`dirname $0`
+programDir=$(readlink -f $programDir)
+parentDir="$(dirname $programDir)"
+programDirBaseName=$(basename $programDir)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+set -x
+
+
+source $(readlink -f $1)
+
+[ "${CONTROL_NODE_PRIVATE_IP}" ]
+[ "${NETWORK_NODES_PRIVATE_IP}" ]
+[ "${COMPUTE_NODES_PRIVATE_IP}" ]
+[ "${STORAGE_NODES_PRIVATE_IP}" ]
+
+
+
+allIpList=`echo "
+${CONTROL_NODE_PRIVATE_IP}
+${NETWORK_NODES_PRIVATE_IP}
+${COMPUTE_NODES_PRIVATE_IP}
+${STORAGE_NODES_PRIVATE_IP}" | sed -e 's/,/\n/g' | sort | uniq `
+
+for IP in ${allIpList}; do
+    ssh root@${IP} 'mkdir -p /tmp/stackube_install'
+    scp ${programDir}/openstack/remove_openstack_from_node.sh root@${IP}:/tmp/stackube_install/
+    scp ${programDir}/lib_tls.sh root@${IP}:/tmp/stackube_install/
+    ssh root@${IP} "/bin/bash /tmp/stackube_install/remove_openstack_from_node.sh"
+done
+
+
+
+exit 0
+