Run etcd and flanneld in a system container

In Fedora Atomic 27 etcd and flanneld are removed from the base image.
Install them as a system containers.

* update docker-storage configuration
* add etcd and flannel tags as labels

Change-Id: I2103c7c3d50f4b68ddc11abff72bc9e3f22839f3
Closes-Bug: #1735381
This commit is contained in:
Spyros Trigazis 2017-11-30 09:53:48 +00:00 committed by Spyros Trigazis
parent ec08641e90
commit d95ba4d1ff
22 changed files with 116 additions and 60 deletions

View File

@ -13,7 +13,7 @@ if is_service_enabled magnum-api magnum-cond; then
echo_summary "Installing magnum"
install_magnum
MAGNUM_GUEST_IMAGE_URL=${MAGNUM_GUEST_IMAGE_URL:-"https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2"}
MAGNUM_GUEST_IMAGE_URL=${MAGNUM_GUEST_IMAGE_URL:-"https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2"}
IMAGE_URLS+=",${MAGNUM_GUEST_IMAGE_URL}"
LIBS_FROM_GIT="${LIBS_FROM_GIT},python-magnumclient"

View File

@ -208,7 +208,7 @@ when installing devstack use::
+--------------------------------------+------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+------------------------------------+--------+
| 0bc132b1-ee91-4bd8-b0fd-19deb57fb39f | Fedora-Atomic-26-20170723.0.x86_64 | active |
| 0bc132b1-ee91-4bd8-b0fd-19deb57fb39f | Fedora-Atomic-27-20180212.2.x86_64 | active |
| 7537bbf2-f1c3-47da-97bb-38c09007e146 | cirros-0.3.5-x86_64-disk | active |
+--------------------------------------+------------------------------------+--------+
@ -253,7 +253,7 @@ Atomic image so the clusters which use this ClusterTemplate will be based on
Fedora Atomic::
openstack coe cluster template create k8s-cluster-template \
--image Fedora-Atomic-26-20170723.0.x86_64 \
--image Fedora-Atomic-27-20180212.2.x86_64 \
--keypair testkey \
--external-network public \
--dns-nameserver 8.8.8.8 \
@ -529,7 +529,7 @@ except for the absence of some Kubernetes-specific arguments and the use of
'swarm' as the COE::
openstack coe cluster template create swarm-cluster-template \
--image Fedora-Atomic-26-20170723.0.x86_64 \
--image Fedora-Atomic-27-20180212.2.x86_64 \
--keypair testkey \
--external-network public \
--dns-nameserver 8.8.8.8 \

View File

@ -116,7 +116,7 @@ and tested by the Magnum team.
.. code-block:: console
$ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2
$ wget https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2
#. Register the image to the Image service setting the ``os_distro`` property
to ``fedora-atomic``:
@ -126,7 +126,7 @@ and tested by the Magnum team.
$ openstack image create \
--disk-format=qcow2 \
--container-format=bare \
--file=Fedora-Atomic-26-20170723.0.x86_64.qcow2\
--file=Fedora-Atomic-27-20180212.2.x86_64.qcow2\
--property os_distro='fedora-atomic' \
fedora-atomic-latest
+------------------+------------------------------------------------------+

View File

@ -338,6 +338,10 @@ the table are linked to more details elsewhere in the user guide.
+---------------------------------------+--------------------+---------------+
| `kube_tag`_ | see below | see below |
+---------------------------------------+--------------------+---------------+
| `etcd_tag`_ | see below | see below |
+---------------------------------------+--------------------+---------------+
| `flannel_tag`_ | see below | see below |
+---------------------------------------+--------------------+---------------+
| `kube_dashboard_enabled`_ | - true | true |
| | - false | |
+---------------------------------------+--------------------+---------------+
@ -1098,6 +1102,8 @@ _`container_infra_prefix`
* docker.io/openstackmagnum/kubernetes-kubelet
* docker.io/openstackmagnum/kubernetes-proxy
* docker.io/openstackmagnum/kubernetes-scheduler
* docker.io/openstackmagnum/etcd
* docker.io/openstackmagnum/flannel
* docker.io/prom/node-exporter:latest
* docker.io/prom/prometheus:latest
* gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
@ -1110,6 +1116,20 @@ _`kube_tag`
If unset, the current Magnum version's default Kubernetes release is
installed.
_`etcd_tag`
This label allows users to select `a specific etcd version,
based on its container tag
<https://hub.docker.com/r/openstackmagnum/etcd/tags/>`_.
If unset, the current Magnum version's a default etcd version.
For queens, v3.2.7
_`flannel_tag`
This label allows users to select `a specific flannel version,
based on its container tag
<https://hub.docker.com/r/openstackmagnum/flannel/tags/>`_.
If unset, the current Magnum version's a default flannel version.
For queens, v0.9.0
_`kube_dashboard_enabled`
This label triggers the deployment of the kubernetes dashboard.
The default value is 1, meaning it will be enabled.

View File

@ -7,10 +7,9 @@
clear_docker_storage () {
# stop docker
systemctl stop docker
systemctl disable docker-storage-setup
# clear storage graph
rm -rf /var/lib/docker/*
# remove current LVs
docker-storage-setup --reset
if [ -f /etc/sysconfig/docker-storage ]; then
sed -i "/^DOCKER_STORAGE_OPTIONS=/ s/=.*/=/" /etc/sysconfig/docker-storage
@ -27,11 +26,7 @@ configure_storage_driver_generic() {
mount -a
fi
sed -i "/^DOCKER_STORAGE_OPTIONS=/ s/=.*/=-s $1/" /etc/sysconfig/docker-storage
local lvname=$(lvdisplay | grep "LV\ Path" | awk '{print $3}')
local pvname=$(pvdisplay | grep "PV\ Name" | awk '{print $3}')
lvextend -r $lvname $pvname
echo "DOCKER_STORAGE_OPTIONS=\"--storage-driver $1\"" > /etc/sysconfig/docker-storage
}
# Configure docker storage with devicemapper using direct LVM
@ -39,7 +34,7 @@ configure_devicemapper () {
clear_docker_storage
echo "GROWROOT=True" > /etc/sysconfig/docker-storage-setup
echo "ROOT_SIZE=5GB" >> /etc/sysconfig/docker-storage-setup
echo "STORAGE_DRIVER=devicemapper" >> /etc/sysconfig/docker-storage-setup
if [ -n "$DOCKER_VOLUME_SIZE" ] && [ "$DOCKER_VOLUME_SIZE" -gt 0 ]; then
@ -48,6 +43,7 @@ configure_devicemapper () {
echo "VG=docker" >> /etc/sysconfig/docker-storage-setup
else
echo "ROOT_SIZE=5GB" >> /etc/sysconfig/docker-storage-setup
echo "DATA_SIZE=95%FREE" >> /etc/sysconfig/docker-storage-setup
fi

View File

@ -34,13 +34,20 @@ if [ -n "$ETCD_VOLUME_SIZE" ] && [ "$ETCD_VOLUME_SIZE" -gt 0 ]; then
fi
_prefix=${CONTAINER_INFRA_PREFIX:-docker.io/openstackmagnum/}
atomic install \
--system-package no \
--system \
--storage ostree \
--name=etcd ${_prefix}etcd:${ETCD_TAG}
if [ -z "$KUBE_NODE_IP" ]; then
# FIXME(yuanying): Set KUBE_NODE_IP correctly
KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
fi
myip="${KUBE_NODE_IP}"
cert_dir="/etc/kubernetes/certs"
cert_dir="/etc/etcd/certs"
protocol="https"
if [ "$TLS_DISABLED" = "True" ]; then

View File

@ -17,30 +17,16 @@ atomic install --storage ostree --system --system-package=no --name=kube-proxy $
CERT_DIR=/etc/kubernetes/certs
PROTOCOL=https
FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
-etcd-certfile $CERT_DIR/proxy.crt \
-etcd-keyfile $CERT_DIR/proxy.key"
ETCD_CURL_OPTIONS="--cacert $CERT_DIR/ca.crt \
--cert $CERT_DIR/proxy.crt --key $CERT_DIR/proxy.key"
ETCD_SERVER_IP=${ETCD_SERVER_IP:-$KUBE_MASTER_IP}
KUBE_PROTOCOL="https"
KUBELET_KUBECONFIG=/etc/kubernetes/kubelet-config.yaml
PROXY_KUBECONFIG=/etc/kubernetes/proxy-config.yaml
FLANNELD_CONFIG=/etc/sysconfig/flanneld
if [ "$TLS_DISABLED" = "True" ]; then
PROTOCOL=http
FLANNEL_OPTIONS=""
ETCD_CURL_OPTIONS=""
KUBE_PROTOCOL="http"
fi
sed -i '/FLANNEL_OPTIONS/'d $FLANNELD_CONFIG
cat >> $FLANNELD_CONFIG <<EOF
FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
EOF
KUBE_MASTER_URI="$KUBE_PROTOCOL://$KUBE_MASTER_IP:$KUBE_API_PORT"
HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
@ -162,9 +148,25 @@ sed -i '
' /etc/kubernetes/proxy
if [ "$NETWORK_DRIVER" = "flannel" ]; then
sed -i '
/^FLANNEL_ETCD_ENDPOINTS=/ s|=.*|="'"$PROTOCOL"'://'"$ETCD_SERVER_IP"':2379"|
' $FLANNELD_CONFIG
atomic install --storage ostree --system --system-package=no \
--name=flanneld ${_prefix}flannel:${FLANNEL_TAG}
if [ "$TLS_DISABLED" = "True" ]; then
FLANNEL_OPTIONS=""
ETCD_CURL_OPTIONS=""
else
FLANNEL_CERT_DIR=/etc/flanneld/certs
FLANNEL_OPTIONS="-etcd-cafile $FLANNEL_CERT_DIR/ca.crt"
FLANNEL_OPTIONS="$FLANNEL_OPTIONS -etcd-certfile $FLANNEL_CERT_DIR/proxy.crt"
FLANNEL_OPTIONS="$FLANNEL_OPTIONS -etcd-keyfile $FLANNEL_CERT_DIR/proxy.key"
ETCD_CURL_OPTIONS="--cacert $FLANNEL_CERT_DIR/ca.crt --cert $FLANNEL_CERT_DIR/proxy.crt --key $FLANNEL_CERT_DIR/proxy.key"
fi
FLANNELD_CONFIG=/etc/sysconfig/flanneld
cat >> $FLANNELD_CONFIG <<EOF
FLANNEL_ETCD_ENDPOINTS="$PROTOCOL://${ETCD_SERVER_IP}:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
EOF
# Make sure etcd has a flannel configuration
. $FLANNELD_CONFIG

View File

@ -141,3 +141,5 @@ chmod 550 "${cert_dir}"
chown -R kube:kube_etcd "${cert_dir}"
chmod 440 ${cert_dir}/kubelet.key
chmod 440 ${cert_dir}/proxy.key
mkdir -p /etc/flanneld/certs
cp ${cert_dir}/* /etc/flanneld/certs

View File

@ -141,3 +141,5 @@ usermod -a -G kube_etcd kube
chmod 550 "${cert_dir}"
chown -R kube:kube_etcd "${cert_dir}"
chmod 440 $SERVER_KEY
mkdir -p /etc/etcd/certs
cp ${cert_dir}/* /etc/etcd/certs

View File

@ -7,25 +7,15 @@ if [ "$NETWORK_DRIVER" != "flannel" ]; then
fi
CERT_DIR=/etc/kubernetes/certs
PROTOCOL=https
FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
-etcd-certfile $CERT_DIR/server.crt \
-etcd-keyfile $CERT_DIR/server.key"
ETCD_CURL_OPTIONS="--cacert $CERT_DIR/ca.crt \
--cert $CERT_DIR/server.crt --key $CERT_DIR/server.key"
FLANNELD_CONFIG=/etc/sysconfig/flanneld
if [ "$TLS_DISABLED" = "True" ]; then
PROTOCOL=http
FLANNEL_OPTIONS=""
ETCD_CURL_OPTIONS=""
fi
sed -i '/FLANNEL_OPTIONS/'d $FLANNELD_CONFIG
cat >> $FLANNELD_CONFIG <<EOF
FLANNEL_OPTIONS="$FLANNEL_OPTIONS"
EOF
. $FLANNELD_CONFIG
FLANNEL_CONFIG_BIN=/usr/local/bin/flannel-config

View File

@ -43,7 +43,7 @@ Requires=flanneld.service
[Service]
Type=oneshot
EnvironmentFile=/run/flannel/subnet.env
EnvironmentFile=/run/flanneld/subnet.env
ExecStart=/usr/local/bin/flannel-docker-bridge
[Install]
@ -79,4 +79,4 @@ chmod 0644 $FLANNEL_DOCKER_BRIDGE_CONF
echo "activating service flanneld"
systemctl enable flanneld
systemctl --no-block start flanneld
systemctl start flanneld

View File

@ -40,6 +40,7 @@ write_files:
NO_PROXY="$NO_PROXY"
WAIT_CURL="$WAIT_CURL"
KUBE_TAG="$KUBE_TAG"
ETCD_TAG="$ETCD_TAG"
KUBE_VERSION="$KUBE_VERSION"
KUBE_DASHBOARD_VERSION="$KUBE_DASHBOARD_VERSION"
TRUSTEE_USER_ID="$TRUSTEE_USER_ID"

View File

@ -37,6 +37,7 @@ write_files:
NO_PROXY="$NO_PROXY"
WAIT_CURL="$WAIT_CURL"
KUBE_TAG="$KUBE_TAG"
FLANNEL_TAG="$FLANNEL_TAG"
KUBE_VERSION="$KUBE_VERSION"
TRUSTEE_USER_ID="$TRUSTEE_USER_ID"
TRUSTEE_PASSWORD="$TRUSTEE_PASSWORD"

View File

@ -2,6 +2,7 @@
. /etc/sysconfig/heat-params
mkdir -p /etc/kubernetes/
KUBE_OS_CLOUD_CONFIG=/etc/kubernetes/kube_openstack_config
cp /etc/pki/tls/certs/ca-bundle.crt /etc/kubernetes/ca-bundle.crt

View File

@ -6,14 +6,14 @@ if [ "$NETWORK_DRIVER" != "flannel" ]; then
exit 0
fi
. /etc/sysconfig/flanneld
FLANNEL_JSON=/etc/sysconfig/flannel-network.json
FLANNELD_CONFIG=/etc/sysconfig/flanneld
sed -i '
/^FLANNEL_ETCD_ENDPOINTS=/ s/=.*/="http:\/\/127.0.0.1:2379"/
' /etc/sysconfig/flanneld
cat > /etc/sysconfig/flanneld <<EOF
FLANNEL_ETCD_ENDPOINTS="http://127.0.0.1:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS=
EOF
# Generate a flannel configuration that we will
# store into etcd using curl.

View File

@ -94,7 +94,8 @@ class K8sFedoraTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
label_list = ['kube_tag', 'container_infra_prefix',
'availability_zone',
'calico_tag', 'calico_cni_tag',
'calico_kube_controllers_tag', 'calico_ipv4pool']
'calico_kube_controllers_tag', 'calico_ipv4pool',
'etcd_tag', 'flannel_tag']
for label in label_list:
label_value = cluster.labels.get(label)
if label_value:

View File

@ -315,6 +315,16 @@ parameters:
description: tag of the k8s containers used to provision the kubernetes cluster
default: v1.9.3
etcd_tag:
type: string
description: tag of the etcd system container
default: v3.2.7
flannel_tag:
type: string
description: tag of the flannel system containers
default: v0.9.0
kube_version:
type: string
description: version of kubernetes used for kubernetes cluster
@ -586,6 +596,7 @@ resources:
no_proxy: {get_param: no_proxy}
kube_tag: {get_param: kube_tag}
kube_version: {get_param: kube_version}
etcd_tag: {get_param: etcd_tag}
kube_dashboard_version: {get_param: kube_dashboard_version}
trustee_user_id: {get_param: trustee_user_id}
trustee_password: {get_param: trustee_password}
@ -665,6 +676,7 @@ resources:
no_proxy: {get_param: no_proxy}
kube_tag: {get_param: kube_tag}
kube_version: {get_param: kube_version}
flannel_tag: {get_param: flannel_tag}
trustee_user_id: {get_param: trustee_user_id}
trustee_username: {get_param: trustee_username}
trustee_password: {get_param: trustee_password}

View File

@ -211,6 +211,10 @@ parameters:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
etcd_tag:
type: string
description: tag of the etcd system container
kube_version:
type: string
description: version of kubernetes used for kubernetes cluster
@ -379,6 +383,7 @@ resources:
"$HTTPS_PROXY": {get_param: https_proxy}
"$NO_PROXY": {get_param: no_proxy}
"$KUBE_TAG": {get_param: kube_tag}
"$ETCD_TAG": {get_param: etcd_tag}
"$KUBE_VERSION": {get_param: kube_version}
"$KUBE_DASHBOARD_VERSION": {get_param: kube_dashboard_version}
"$WAIT_CURL": {get_attr: [master_wait_handle, curl_cli]}
@ -459,12 +464,6 @@ resources:
group: ungrouped
config: {get_file: ../../common/templates/kubernetes/fragments/enable-services-master.sh}
network_service:
type: OS::Heat::SoftwareConfig
properties:
group: ungrouped
config: {get_file: ../../common/templates/kubernetes/fragments/network-service.sh}
kube_apiserver_to_kubelet_role:
type: OS::Heat::SoftwareConfig
properties:
@ -516,15 +515,14 @@ resources:
- config: {get_resource: write_heat_params}
- config: {get_resource: configure_etcd}
- config: {get_resource: write_kube_os_config}
- config: {get_resource: make_cert}
- config: {get_resource: configure_docker_storage}
- config: {get_resource: configure_kubernetes}
- config: {get_resource: make_cert}
- config: {get_resource: add_proxy}
- config: {get_resource: start_container_agent}
- config: {get_resource: enable_services}
- config: {get_resource: write_network_config}
- config: {get_resource: network_config_service}
- config: {get_resource: network_service}
- config: {get_resource: kube_apiserver_to_kubelet_role}
- config: {get_resource: core_dns_service}
- config: {get_resource: kube_ui_service}

View File

@ -175,6 +175,10 @@ parameters:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
flannel_tag:
type: string
description: tag of the flannel system containers
kube_version:
type: string
description: version of kubernetes used for kubernetes cluster
@ -300,6 +304,7 @@ resources:
$HTTPS_PROXY: {get_param: https_proxy}
$NO_PROXY: {get_param: no_proxy}
$KUBE_TAG: {get_param: kube_tag}
$FLANNEL_TAG: {get_param: flannel_tag}
$KUBE_VERSION: {get_param: kube_version}
$WAIT_CURL: {get_attr: [minion_wait_handle, curl_cli]}
$TRUSTEE_USER_ID: {get_param: trustee_user_id}

View File

@ -85,8 +85,8 @@ elif [ "${coe}${special}" = "k8s-ironic" ]; then
export DEVSTACK_LOCAL_CONFIG+=$'\n'"IRONIC_VM_SPECS_DISK=10"
export DEVSTACK_LOCAL_CONFIG+=$'\n'"IRONIC_VM_EPHEMERAL_DISK=5"
else
export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_GUEST_IMAGE_URL='${NODEPOOL_ATOMIC_MIRROR}/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2'"
export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_IMAGE_NAME='Fedora-Atomic-26-20170723.0.x86_64'"
export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_GUEST_IMAGE_URL='${NODEPOOL_ATOMIC_MIRROR}/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2'"
export DEVSTACK_LOCAL_CONFIG+=$'\n'"MAGNUM_IMAGE_NAME='Fedora-Atomic-27-20180212.2.x86_64'"
fi
# Enable magnum plugin in the last step

View File

@ -271,6 +271,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
etcd_volume_size = mock_cluster.labels.get(
'etcd_volume_size')
kube_tag = mock_cluster.labels.get('kube_tag')
etcd_tag = mock_cluster.labels.get('etcd_tag')
flannel_tag = mock_cluster.labels.get('flannel_tag')
container_infra_prefix = mock_cluster.labels.get(
'container_infra_prefix')
availability_zone = mock_cluster.labels.get(
@ -312,6 +314,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
'magnum_url': mock_osc.magnum_url.return_value,
'region_name': mock_osc.cinder_region_name.return_value,
'kube_tag': kube_tag,
'etcd_tag': etcd_tag,
'flannel_tag': flannel_tag,
'container_infra_prefix': container_infra_prefix,
'nodes_affinity_policy': 'soft-anti-affinity',
'availability_zone': availability_zone,
@ -379,6 +383,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
etcd_volume_size = mock_cluster.labels.get(
'etcd_volume_size')
kube_tag = mock_cluster.labels.get('kube_tag')
etcd_tag = mock_cluster.labels.get('etcd_tag')
flannel_tag = mock_cluster.labels.get('flannel_tag')
container_infra_prefix = mock_cluster.labels.get(
'container_infra_prefix')
availability_zone = mock_cluster.labels.get(
@ -422,6 +428,8 @@ class AtomicK8sTemplateDefinitionTestCase(BaseTemplateDefinitionTestCase):
'loadbalancing_protocol': 'HTTP',
'kubernetes_port': 8080,
'kube_tag': kube_tag,
'etcd_tag': etcd_tag,
'flannel_tag': flannel_tag,
'container_infra_prefix': container_infra_prefix,
'nodes_affinity_policy': 'soft-anti-affinity',
'availability_zone': availability_zone,

View File

@ -0,0 +1,10 @@
---
features:
- |
Update k8s_fedora_atomic driver to the latest Fedora Atomic 27 release
and run etcd and flanneld in system containers which are removed from
the base OS.
upgrade:
- |
New clusters should be created with kube_tag=v1.9.3 or later. v1.9.3 is
the default version in the queens release.