Merge "Make cinder volume optional"

This commit is contained in:
Jenkins 2016-11-10 18:25:56 +00:00 committed by Gerrit Code Review
commit 0c1615ef1c
25 changed files with 232 additions and 121 deletions

View File

@ -206,11 +206,11 @@ They are loosely grouped as: mandatory, infrastructure, COE specific.
is 'None'. is 'None'.
--docker-volume-size \<docker-volume-size\> --docker-volume-size \<docker-volume-size\>
The size in GB for the local storage on each server for the Docker If specified, container images will be stored in a cinder volume of the
daemon to cache the images and host the containers. Cinder volumes specified size in GB. Each cluster node will have a volume attached of
provide the storage. The default is 25 GB. For the 'devicemapper' the above size. If not specified, images will be stored in the compute
storage driver, the minimum value is 3GB. For the 'overlay' storage instance's local disk. For the 'devicemapper' storage driver, the minimum
driver, the minimum value is 1GB. value is 3GB. For the 'overlay' storage driver, the minimum value is 1GB.
--docker-storage-driver \<docker-storage-driver\> --docker-storage-driver \<docker-storage-driver\>
The name of a driver to manage the storage for the images and the The name of a driver to manage the storage for the images and the
@ -359,8 +359,8 @@ Network
needed. needed.
Storage Storage
Cinder provides the block storage that is used for both hosting the Cinder provides the block storage that can be used to host the
containers as well as persistent storage for the containers. containers and as persistent storage for the containers.
Security Security
Barbican provides the storage of secrets such as certificates used Barbican provides the storage of secrets such as certificates used
@ -981,14 +981,8 @@ Volume driver (volume-driver)
Storage driver (docker-storage-driver) Storage driver (docker-storage-driver)
Specified in the ClusterTemplate to select the Docker storage driver. The Specified in the ClusterTemplate to select the Docker storage driver. The
supported storage drivers are 'devicemapper' and 'overlay', with supported storage drivers are 'devicemapper' and 'overlay', with
'devicemapper' being the default. You may get better performance with 'devicemapper' being the default. Refer to the `Storage`_ section for more
the overlay driver depending on your use patterns, with the requirement details.
that SELinux must be disabled inside the containers, although it still runs
in enforcing mode on the cluster servers. Magnum will create a Cinder volume
for each node, mount it on the node and configure it as a logical
volume named 'docker'. The Docker daemon will run the selected device
driver to manage this logical volume and host the container writable
layer there. Refer to the `Storage`_ section for more details.
Image (image-id) Image (image-id)
Specified in the ClusterTemplate to indicate the image to boot the servers. Specified in the ClusterTemplate to indicate the image to boot the servers.
@ -1126,15 +1120,8 @@ Volume driver (volume-driver)
Storage driver (docker-storage-driver) Storage driver (docker-storage-driver)
Specified in the ClusterTemplate to select the Docker storage driver. The Specified in the ClusterTemplate to select the Docker storage driver. The
supported storage driver are 'devicemapper' and 'overlay', with supported storage driver are 'devicemapper' and 'overlay', with
'devicemapper' being the default. You may get better performance with 'devicemapper' being the default. Refer to the `Storage`_ section for more
the 'overlay' driver depending on your use patterns, with the requirement details.
that SELinux must be disabled inside the containers, although it still runs
in enforcing mode on the cluster servers. Magnum will create a Cinder volume
for each node and attach it as a device. Then depending on the driver,
additional configuration is performed to make the volume available to
the particular driver. For instance, 'devicemapper' uses LVM; therefore
Magnum will create physical volume and logical volume using the attached
device. Refer to the `Storage`_ section for more details.
Image (image-id) Image (image-id)
Specified in the ClusterTemplate to indicate the image to boot the servers Specified in the ClusterTemplate to indicate the image to boot the servers
@ -2132,25 +2119,32 @@ configured in the Docker daemon through a number of storage options.
When the container is removed, the storage allocated to the particular When the container is removed, the storage allocated to the particular
container is also deleted. container is also deleted.
To manage this space in a flexible manner independent of the Nova Magnum can manage the containers' filesystem in two ways, storing them
instance flavor, Magnum creates a separate Cinder block volume for each on the local disk of the compute instances or in a separate Cinder block
node in the cluster, mounts it to the node and configures it to be used as volume for each node in the cluster, mounts it to the node and
ephemeral storage. Users can specify the size of the Cinder volume with configures it to be used as ephemeral storage. Users can specify the
the ClusterTemplate attribute 'docker-volume-size'. The default size is 5GB. size of the Cinder volume with the ClusterTemplate attribute
Currently the block size is fixed at cluster creation time, but future 'docker-volume-size'. Currently the block size is fixed at cluster
lifecycle operations may allow modifying the block size during the creation time, but future lifecycle operations may allow modifying the
life of the cluster. block size during the life of the cluster.
To use the Cinder block storage, there is a number of Docker Both local disk and the Cinder block storage can be used with a number
storage drivers available. Only 'devicemapper' is supported as the of Docker storage drivers available.
storage driver but other drivers such as 'OverlayFS' are being
considered. There are important trade-off between the choices
for the storage drivers that should be considered. For instance,
'OperlayFS' may offer better performance, but it may not support
the filesystem metadata needed to use SELinux, which is required
to support strong isolation between containers running in the same
cluster. Using the 'devicemapper' driver does allow the use of SELinux.
* 'devicemapper': When used with a dedicated Cinder volume it is
configured using direct-lvm and offers very good performance. If it's
used with the compute instance's local disk uses a loopback device
offering poor performance and it's not recommended for production
environments. Using the 'devicemapper' driver does allow the use of
SELinux.
* 'overlay' When used with a dedicated Cinder volume offers as good
or better performance than devicemapper. If used on the local disk of
the compute instance (especially with high IOPS drives) you can get
significant performance gains. However, for kernel versions less than
4.9, SELinux must be disabled inside the containers resulting in worse
container isolation, although it still runs in enforcing mode on the
cluster compute instances.
Persistent storage Persistent storage
------------------ ------------------

View File

@ -80,17 +80,19 @@ class K8sFedoraTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
def get_env_files(self, cluster_template): def get_env_files(self, cluster_template):
env_files = [] env_files = []
if cluster_template.master_lb_enabled:
env_files.append(
template_def.COMMON_ENV_PATH + 'with_master_lb.yaml')
else:
env_files.append(
template_def.COMMON_ENV_PATH + 'no_master_lb.yaml')
if cluster_template.floating_ip_enabled:
env_files.append(
template_def.COMMON_ENV_PATH + 'enable_floating_ip.yaml')
else:
env_files.append(
template_def.COMMON_ENV_PATH + 'disable_floating_ip.yaml')
return env_files if cluster_template.docker_volume_size is None:
env_files.append('no_volume.yaml')
else:
env_files.append('with_volume.yaml')
if cluster_template.master_lb_enabled:
env_files.append('with_master_lb.yaml')
else:
env_files.append('no_master_lb.yaml')
if cluster_template.floating_ip_enabled:
env_files.append('enable_floating_ip.yaml')
else:
env_files.append('disable_floating_ip.yaml')
return [template_def.COMMON_ENV_PATH + ef for ef in env_files]

View File

@ -113,7 +113,16 @@ class SwarmFedoraTemplateDefinition(template_def.BaseTemplateDefinition):
**kwargs) **kwargs)
def get_env_files(self, cluster_template): def get_env_files(self, cluster_template):
if cluster_template.master_lb_enabled: env_files = []
return [template_def.COMMON_ENV_PATH + 'with_master_lb.yaml']
if cluster_template.docker_volume_size is None:
env_files.append('no_volume.yaml')
else: else:
return [template_def.COMMON_ENV_PATH + 'no_master_lb.yaml'] env_files.append('with_volume.yaml')
if cluster_template.master_lb_enabled:
env_files.append('with_master_lb.yaml')
else:
env_files.append('no_master_lb.yaml')
return [template_def.COMMON_ENV_PATH + ef for ef in env_files]

View File

@ -0,0 +1,4 @@
# Environment file to NOT use a cinder volume to store containers
resource_registry:
"Magnum::Optional::Cinder::Volume": "OS::Heat::None"
"Magnum::Optional::Cinder::VolumeAttachment": "OS::Heat::None"

View File

@ -0,0 +1,4 @@
# Environment file to use a cinder volume to store containers
resource_registry:
"Magnum::Optional::Cinder::Volume": "OS::Cinder::Volume"
"Magnum::Optional::Cinder::VolumeAttachment": "OS::Cinder::VolumeAttachment"

View File

@ -2,30 +2,32 @@
. /etc/sysconfig/heat-params . /etc/sysconfig/heat-params
if [ "$ENABLE_CINDER" == "False" ]; then if [ -n "$DOCKER_VOLUME_SIZE" ] && [ "$DOCKER_VOLUME_SIZE" -gt 0 ]; then
# FIXME(yuanying): Use ephemeral disk for docker storage if [ "$ENABLE_CINDER" == "False" ]; then
# Currently Ironic doesn't support cinder volumes, # FIXME(yuanying): Use ephemeral disk for docker storage
# so we must use preserved ephemeral disk instead of a cinder volume. # Currently Ironic doesn't support cinder volumes,
device_path=$(readlink -f /dev/disk/by-label/ephemeral0) # so we must use preserved ephemeral disk instead of a cinder volume.
else device_path=$(readlink -f /dev/disk/by-label/ephemeral0)
attempts=60 else
while [ ${attempts} -gt 0 ]; do attempts=60
device_name=$(ls /dev/disk/by-id | grep ${DOCKER_VOLUME:0:20}$) while [ ${attempts} -gt 0 ]; do
if [ -n "${device_name}" ]; then device_name=$(ls /dev/disk/by-id | grep ${DOCKER_VOLUME:0:20}$)
break if [ -n "${device_name}" ]; then
fi break
echo "waiting for disk device" fi
sleep 0.5 echo "waiting for disk device"
udevadm trigger sleep 0.5
let attempts-- udevadm trigger
done let attempts--
done
if [ -z "${device_name}" ]; then if [ -z "${device_name}" ]; then
echo "ERROR: disk device does not exist" >&2 echo "ERROR: disk device does not exist" >&2
exit 1 exit 1
fi
device_path=/dev/disk/by-id/${device_name}
fi fi
device_path=/dev/disk/by-id/${device_name}
fi fi
$configure_docker_storage_driver $configure_docker_storage_driver

View File

@ -15,9 +15,11 @@ configure_overlay () {
rm -rf /var/lib/docker/* rm -rf /var/lib/docker/*
mkfs.xfs -f ${device_path} if [ -n "$DOCKER_VOLUME_SIZE" ] && [ "$DOCKER_VOLUME_SIZE" -gt 0 ]; then
echo "${device_path} /var/lib/docker xfs defaults 0 0" >> /etc/fstab mkfs.xfs -f ${device_path}
mount -a echo "${device_path} /var/lib/docker xfs defaults 0 0" >> /etc/fstab
mount -a
fi
echo "STORAGE_DRIVER=overlay" > /etc/sysconfig/docker-storage-setup echo "STORAGE_DRIVER=overlay" > /etc/sysconfig/docker-storage-setup
@ -31,8 +33,10 @@ configure_overlay () {
configure_devicemapper () { configure_devicemapper () {
clear_docker_storage_congiguration clear_docker_storage_congiguration
pvcreate -f ${device_path} if [ -n "$DOCKER_VOLUME_SIZE" ] && [ "$DOCKER_VOLUME_SIZE" -gt 0 ]; then
vgcreate docker ${device_path} pvcreate -f ${device_path}
vgcreate docker ${device_path}
echo "VG=docker" > /etc/sysconfig/docker-storage-setup echo "VG=docker" > /etc/sysconfig/docker-storage-setup
fi
} }

View File

@ -13,6 +13,7 @@ write_files:
KUBE_ALLOW_PRIV="$KUBE_ALLOW_PRIV" KUBE_ALLOW_PRIV="$KUBE_ALLOW_PRIV"
ENABLE_CINDER="$ENABLE_CINDER" ENABLE_CINDER="$ENABLE_CINDER"
DOCKER_VOLUME="$DOCKER_VOLUME" DOCKER_VOLUME="$DOCKER_VOLUME"
DOCKER_VOLUME_SIZE="$DOCKER_VOLUME_SIZE"
DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER" DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER"
NETWORK_DRIVER="$NETWORK_DRIVER" NETWORK_DRIVER="$NETWORK_DRIVER"
FLANNEL_NETWORK_CIDR="$FLANNEL_NETWORK_CIDR" FLANNEL_NETWORK_CIDR="$FLANNEL_NETWORK_CIDR"

View File

@ -13,6 +13,7 @@ write_files:
ETCD_SERVER_IP="$ETCD_SERVER_IP" ETCD_SERVER_IP="$ETCD_SERVER_IP"
ENABLE_CINDER="$ENABLE_CINDER" ENABLE_CINDER="$ENABLE_CINDER"
DOCKER_VOLUME="$DOCKER_VOLUME" DOCKER_VOLUME="$DOCKER_VOLUME"
DOCKER_VOLUME_SIZE="$DOCKER_VOLUME_SIZE"
DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER" DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER"
NETWORK_DRIVER="$NETWORK_DRIVER" NETWORK_DRIVER="$NETWORK_DRIVER"
REGISTRY_ENABLED="$REGISTRY_ENABLED" REGISTRY_ENABLED="$REGISTRY_ENABLED"

View File

@ -10,6 +10,7 @@ write_files:
WAIT_CURL="$WAIT_CURL" WAIT_CURL="$WAIT_CURL"
ETCD_DISCOVERY_URL="$ETCD_DISCOVERY_URL" ETCD_DISCOVERY_URL="$ETCD_DISCOVERY_URL"
DOCKER_VOLUME="$DOCKER_VOLUME" DOCKER_VOLUME="$DOCKER_VOLUME"
DOCKER_VOLUME_SIZE="$DOCKER_VOLUME_SIZE"
DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER" DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER"
HTTP_PROXY="$HTTP_PROXY" HTTP_PROXY="$HTTP_PROXY"
HTTPS_PROXY="$HTTPS_PROXY" HTTPS_PROXY="$HTTPS_PROXY"

View File

@ -9,6 +9,7 @@ write_files:
WAIT_HANDLE_TOKEN="$WAIT_HANDLE_TOKEN" WAIT_HANDLE_TOKEN="$WAIT_HANDLE_TOKEN"
WAIT_CURL="$WAIT_CURL" WAIT_CURL="$WAIT_CURL"
DOCKER_VOLUME="$DOCKER_VOLUME" DOCKER_VOLUME="$DOCKER_VOLUME"
DOCKER_VOLUME_SIZE="$DOCKER_VOLUME_SIZE"
DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER" DOCKER_STORAGE_DRIVER="$DOCKER_STORAGE_DRIVER"
HTTP_PROXY="$HTTP_PROXY" HTTP_PROXY="$HTTP_PROXY"
HTTPS_PROXY="$HTTPS_PROXY" HTTPS_PROXY="$HTTPS_PROXY"

View File

@ -92,7 +92,7 @@ parameters:
description: > description: >
size of a cinder volume to allocate to docker for container/image size of a cinder volume to allocate to docker for container/image
storage storage
default: 25 default: 0
docker_storage_driver: docker_storage_driver:
type: string type: string

View File

@ -230,6 +230,7 @@ resources:
"$KUBE_NODE_IP": {get_attr: [kube_master_eth0, fixed_ips, 0, ip_address]} "$KUBE_NODE_IP": {get_attr: [kube_master_eth0, fixed_ips, 0, ip_address]}
"$KUBE_ALLOW_PRIV": {get_param: kube_allow_priv} "$KUBE_ALLOW_PRIV": {get_param: kube_allow_priv}
"$DOCKER_VOLUME": {get_resource: docker_volume} "$DOCKER_VOLUME": {get_resource: docker_volume}
"$DOCKER_VOLUME_SIZE": {get_param: docker_volume_size}
"$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver} "$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver}
"$NETWORK_DRIVER": {get_param: network_driver} "$NETWORK_DRIVER": {get_param: network_driver}
"$FLANNEL_NETWORK_CIDR": {get_param: flannel_network_cidr} "$FLANNEL_NETWORK_CIDR": {get_param: flannel_network_cidr}
@ -442,12 +443,12 @@ resources:
# #
docker_volume: docker_volume:
type: OS::Cinder::Volume type: Magnum::Optional::Cinder::Volume
properties: properties:
size: {get_param: docker_volume_size} size: {get_param: docker_volume_size}
docker_volume_attach: docker_volume_attach:
type: OS::Cinder::VolumeAttachment type: Magnum::Optional::Cinder::VolumeAttachment
properties: properties:
instance_uuid: {get_resource: kube_master} instance_uuid: {get_resource: kube_master}
volume_id: {get_resource: docker_volume} volume_id: {get_resource: docker_volume}

View File

@ -227,6 +227,7 @@ resources:
$KUBE_NODE_IP: {get_attr: [kube_minion_eth0, fixed_ips, 0, ip_address]} $KUBE_NODE_IP: {get_attr: [kube_minion_eth0, fixed_ips, 0, ip_address]}
$ETCD_SERVER_IP: {get_param: etcd_server_ip} $ETCD_SERVER_IP: {get_param: etcd_server_ip}
$DOCKER_VOLUME: {get_resource: docker_volume} $DOCKER_VOLUME: {get_resource: docker_volume}
$DOCKER_VOLUME_SIZE: {get_param: docker_volume_size}
$DOCKER_STORAGE_DRIVER: {get_param: docker_storage_driver} $DOCKER_STORAGE_DRIVER: {get_param: docker_storage_driver}
$NETWORK_DRIVER: {get_param: network_driver} $NETWORK_DRIVER: {get_param: network_driver}
$REGISTRY_ENABLED: {get_param: registry_enabled} $REGISTRY_ENABLED: {get_param: registry_enabled}
@ -410,12 +411,12 @@ resources:
# #
docker_volume: docker_volume:
type: OS::Cinder::Volume type: Magnum::Optional::Cinder::Volume
properties: properties:
size: {get_param: docker_volume_size} size: {get_param: docker_volume_size}
docker_volume_attach: docker_volume_attach:
type: OS::Cinder::VolumeAttachment type: Magnum::Optional::Cinder::VolumeAttachment
properties: properties:
instance_uuid: {get_resource: kube-minion} instance_uuid: {get_resource: kube-minion}
volume_id: {get_resource: docker_volume} volume_id: {get_resource: docker_volume}

View File

@ -100,7 +100,7 @@ parameters:
description: > description: >
size of a cinder volume to allocate to docker for container/image size of a cinder volume to allocate to docker for container/image
storage storage
default: 25 default: 0
docker_storage_driver: docker_storage_driver:
type: string type: string
@ -430,6 +430,7 @@ resources:
master_flavor: {get_param: master_flavor} master_flavor: {get_param: master_flavor}
external_network: {get_param: external_network} external_network: {get_param: external_network}
kube_allow_priv: {get_param: kube_allow_priv} kube_allow_priv: {get_param: kube_allow_priv}
docker_volume_size: {get_param: docker_volume_size}
docker_storage_driver: {get_param: docker_storage_driver} docker_storage_driver: {get_param: docker_storage_driver}
wait_condition_timeout: {get_param: wait_condition_timeout} wait_condition_timeout: {get_param: wait_condition_timeout}
network_driver: {get_param: network_driver} network_driver: {get_param: network_driver}
@ -486,6 +487,7 @@ resources:
etcd_server_ip: {get_attr: [etcd_address_switch, private_ip]} etcd_server_ip: {get_attr: [etcd_address_switch, private_ip]}
external_network: {get_param: external_network} external_network: {get_param: external_network}
kube_allow_priv: {get_param: kube_allow_priv} kube_allow_priv: {get_param: kube_allow_priv}
docker_volume_size: {get_param: docker_volume_size}
docker_storage_driver: {get_param: docker_storage_driver} docker_storage_driver: {get_param: docker_storage_driver}
wait_condition_timeout: {get_param: wait_condition_timeout} wait_condition_timeout: {get_param: wait_condition_timeout}
registry_enabled: {get_param: registry_enabled} registry_enabled: {get_param: registry_enabled}

View File

@ -35,6 +35,12 @@ parameters:
constraints: constraints:
- allowed_values: ["true", "false"] - allowed_values: ["true", "false"]
docker_volume_size:
type: number
description: >
size of a cinder volume to allocate to docker for container/image
storage
docker_storage_driver: docker_storage_driver:
type: string type: string
description: docker storage driver name description: docker storage driver name
@ -222,6 +228,7 @@ resources:
"$KUBE_API_PORT": {get_param: kubernetes_port} "$KUBE_API_PORT": {get_param: kubernetes_port}
"$KUBE_ALLOW_PRIV": {get_param: kube_allow_priv} "$KUBE_ALLOW_PRIV": {get_param: kube_allow_priv}
"$DOCKER_VOLUME": 'None' "$DOCKER_VOLUME": 'None'
"$DOCKER_VOLUME_SIZE": {get_param: docker_volume_size}
"$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver} "$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver}
"$NETWORK_DRIVER": {get_param: network_driver} "$NETWORK_DRIVER": {get_param: network_driver}
"$FLANNEL_NETWORK_CIDR": {get_param: flannel_network_cidr} "$FLANNEL_NETWORK_CIDR": {get_param: flannel_network_cidr}

View File

@ -30,6 +30,12 @@ parameters:
constraints: constraints:
- allowed_values: ["true", "false"] - allowed_values: ["true", "false"]
docker_volume_size:
type: number
description: >
size of a cinder volume to allocate to docker for container/image
storage
docker_storage_driver: docker_storage_driver:
type: string type: string
description: docker storage driver name description: docker storage driver name
@ -219,6 +225,7 @@ resources:
$KUBE_API_PORT: {get_param: kubernetes_port} $KUBE_API_PORT: {get_param: kubernetes_port}
$ETCD_SERVER_IP: {get_param: etcd_server_ip} $ETCD_SERVER_IP: {get_param: etcd_server_ip}
$DOCKER_VOLUME: 'None' $DOCKER_VOLUME: 'None'
$DOCKER_VOLUME_SIZE: {get_param: docker_volume_size}
$DOCKER_STORAGE_DRIVER: {get_param: docker_storage_driver} $DOCKER_STORAGE_DRIVER: {get_param: docker_storage_driver}
$NETWORK_DRIVER: {get_param: network_driver} $NETWORK_DRIVER: {get_param: network_driver}
$REGISTRY_ENABLED: {get_param: registry_enabled} $REGISTRY_ENABLED: {get_param: registry_enabled}

View File

@ -118,7 +118,7 @@ parameters:
description: > description: >
size of a cinder volume to allocate to docker for container/image size of a cinder volume to allocate to docker for container/image
storage storage
default: 25 default: 0
docker_storage_driver: docker_storage_driver:
type: string type: string

View File

@ -211,6 +211,7 @@ resources:
"$WAIT_HANDLE_TOKEN": {get_attr: [master_wait_handle, token]} "$WAIT_HANDLE_TOKEN": {get_attr: [master_wait_handle, token]}
"$WAIT_CURL": {get_attr: [master_wait_handle, curl_cli]} "$WAIT_CURL": {get_attr: [master_wait_handle, curl_cli]}
"$DOCKER_VOLUME": {get_resource: docker_volume} "$DOCKER_VOLUME": {get_resource: docker_volume}
"$DOCKER_VOLUME_SIZE": {get_param: docker_volume_size}
"$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver} "$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver}
"$ETCD_DISCOVERY_URL": {get_param: discovery_url} "$ETCD_DISCOVERY_URL": {get_param: discovery_url}
"$HTTP_PROXY": {get_param: http_proxy} "$HTTP_PROXY": {get_param: http_proxy}
@ -445,12 +446,12 @@ resources:
# #
docker_volume: docker_volume:
type: OS::Cinder::Volume type: Magnum::Optional::Cinder::Volume
properties: properties:
size: {get_param: docker_volume_size} size: {get_param: docker_volume_size}
docker_volume_attach: docker_volume_attach:
type: OS::Cinder::VolumeAttachment type: Magnum::Optional::Cinder::VolumeAttachment
properties: properties:
instance_uuid: {get_resource: swarm_master} instance_uuid: {get_resource: swarm_master}
volume_id: {get_resource: docker_volume} volume_id: {get_resource: docker_volume}

View File

@ -189,6 +189,7 @@ resources:
"$WAIT_HANDLE_TOKEN": {get_attr: [node_wait_handle, token]} "$WAIT_HANDLE_TOKEN": {get_attr: [node_wait_handle, token]}
"$WAIT_CURL": {get_attr: [node_wait_handle, curl_cli]} "$WAIT_CURL": {get_attr: [node_wait_handle, curl_cli]}
"$DOCKER_VOLUME": {get_resource: docker_volume} "$DOCKER_VOLUME": {get_resource: docker_volume}
"$DOCKER_VOLUME_SIZE": {get_param: docker_volume_size}
"$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver} "$DOCKER_STORAGE_DRIVER": {get_param: docker_storage_driver}
"$HTTP_PROXY": {get_param: http_proxy} "$HTTP_PROXY": {get_param: http_proxy}
"$HTTPS_PROXY": {get_param: https_proxy} "$HTTPS_PROXY": {get_param: https_proxy}
@ -385,12 +386,12 @@ resources:
# #
docker_volume: docker_volume:
type: OS::Cinder::Volume type: Magnum::Optional::Cinder::Volume
properties: properties:
size: {get_param: docker_volume_size} size: {get_param: docker_volume_size}
docker_volume_attach: docker_volume_attach:
type: OS::Cinder::VolumeAttachment type: Magnum::Optional::Cinder::VolumeAttachment
properties: properties:
instance_uuid: {get_resource: swarm_node} instance_uuid: {get_resource: swarm_node}
volume_id: {get_resource: docker_volume} volume_id: {get_resource: docker_volume}

View File

@ -545,7 +545,6 @@ class TestPost(api_base.FunctionalTest):
self._create_baymodel_raises_app_error(coe='osomatsu') self._create_baymodel_raises_app_error(coe='osomatsu')
def test_create_baymodel_with_invalid_docker_volume_size(self): def test_create_baymodel_with_invalid_docker_volume_size(self):
self._create_baymodel_raises_app_error(docker_volume_size=0)
self._create_baymodel_raises_app_error(docker_volume_size=-1) self._create_baymodel_raises_app_error(docker_volume_size=-1)
self._create_baymodel_raises_app_error( self._create_baymodel_raises_app_error(
docker_volume_size=1, docker_volume_size=1,

View File

@ -604,7 +604,6 @@ class TestPost(api_base.FunctionalTest):
self._create_model_raises_app_error(coe='osomatsu') self._create_model_raises_app_error(coe='osomatsu')
def test_create_cluster_template_with_invalid_docker_volume_size(self): def test_create_cluster_template_with_invalid_docker_volume_size(self):
self._create_model_raises_app_error(docker_volume_size=0)
self._create_model_raises_app_error(docker_volume_size=-1) self._create_model_raises_app_error(docker_volume_size=-1)
self._create_model_raises_app_error( self._create_model_raises_app_error(
docker_volume_size=1, docker_volume_size=1,

View File

@ -192,7 +192,8 @@ class TestClusterConductorWithK8s(base.TestCase):
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml', ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml',
'../../common/templates/environments/disable_floating_ip.yaml'], '../../common/templates/environments/disable_floating_ip.yaml'],
env_files) env_files)
@ -267,7 +268,75 @@ class TestClusterConductorWithK8s(base.TestCase):
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml', ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml',
'../../common/templates/environments/disable_floating_ip.yaml'],
env_files)
@patch('requests.get')
@patch('magnum.objects.ClusterTemplate.get_by_uuid')
@patch('magnum.drivers.common.driver.Driver.get_driver')
def test_extract_template_definition_only_required(
self,
mock_driver,
mock_objects_cluster_template_get_by_uuid,
mock_get):
not_required = ['image_id', 'flavor_id', 'dns_nameserver',
'docker_volume_size', 'fixed_network', 'http_proxy',
'https_proxy', 'no_proxy', 'network_driver',
'master_flavor_id', 'docker_storage_driver',
'volume_driver']
for key in not_required:
self.cluster_template_dict[key] = None
self.cluster_dict['discovery_url'] = 'https://discovery.etcd.io/test'
cluster_template = objects.ClusterTemplate(
self.context, **self.cluster_template_dict)
mock_objects_cluster_template_get_by_uuid.return_value = \
cluster_template
expected_result = str('{"action":"get","node":{"key":"test","value":'
'"1","modifiedIndex":10,"createdIndex":10}}')
mock_resp = mock.MagicMock()
mock_resp.text = expected_result
mock_get.return_value = mock_resp
mock_driver.return_value = k8s_dr.Driver()
cluster = objects.Cluster(self.context, **self.cluster_dict)
(template_path,
definition,
env_files) = driver._extract_template_definition(self.context,
cluster)
expected = {
'auth_url': 'http://192.168.10.10:5000/v3',
'cluster_uuid': '5d12f6fd-a196-4bf0-ae4c-1f639a523a52',
'discovery_url': 'https://discovery.etcd.io/test',
'external_network': 'external_network_id',
'flannel_backend': 'vxlan',
'flannel_network_cidr': '10.101.0.0/16',
'flannel_network_subnetlen': '26',
'insecure_registry_url': '10.0.0.1:5000',
'kube_version': 'fake-version',
'magnum_url': 'http://127.0.0.1:9511/v1',
'number_of_masters': 1,
'number_of_minions': 1,
'region_name': 'RegionOne',
'registry_enabled': False,
'ssh_key_name': 'keypair_id',
'tenant_name': 'fake_tenant',
'tls_disabled': False,
'trust_id': 'bd11efc5-d4e2-4dac-bbce-25e348ddf7de',
'trustee_domain_id': 'trustee_domain_id',
'trustee_password': 'fake_trustee_password',
'trustee_user_id': '7b489f04-b458-4541-8179-6a48a553e656',
'trustee_username': 'fake_trustee',
'username': 'fake_user'
}
self.assertEqual(expected, definition)
self.assertEqual(
['../../common/templates/environments/no_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml',
'../../common/templates/environments/disable_floating_ip.yaml'], '../../common/templates/environments/disable_floating_ip.yaml'],
env_files) env_files)
@ -438,21 +507,6 @@ class TestClusterConductorWithK8s(base.TestCase):
mock_get, mock_get,
missing_attr='flavor_id') missing_attr='flavor_id')
@patch('requests.get')
@patch('magnum.objects.ClusterTemplate.get_by_uuid')
@patch('magnum.drivers.common.driver.Driver.get_driver')
def test_extract_template_definition_without_docker_volume_size(
self,
mock_driver,
mock_objects_cluster_template_get_by_uuid,
mock_get):
mock_driver.return_value = k8s_dr.Driver()
self._test_extract_template_definition(
mock_driver,
mock_objects_cluster_template_get_by_uuid,
mock_get,
missing_attr='docker_volume_size')
@patch('requests.get') @patch('requests.get')
@patch('magnum.objects.ClusterTemplate.get_by_uuid') @patch('magnum.objects.ClusterTemplate.get_by_uuid')
@patch('magnum.drivers.common.driver.Driver.get_driver') @patch('magnum.drivers.common.driver.Driver.get_driver')
@ -594,7 +648,8 @@ class TestClusterConductorWithK8s(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml', ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml',
'../../common/templates/environments/disable_floating_ip.yaml'], '../../common/templates/environments/disable_floating_ip.yaml'],
env_files) env_files)
reqget.assert_called_once_with('http://etcd/test?size=1') reqget.assert_called_once_with('http://etcd/test?size=1')

View File

@ -146,7 +146,8 @@ class TestClusterConductorWithSwarm(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml'], ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml'],
env_files) env_files)
@patch('requests.get') @patch('requests.get')
@ -217,7 +218,8 @@ class TestClusterConductorWithSwarm(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml'], ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml'],
env_files) env_files)
@patch('requests.get') @patch('requests.get')
@ -280,7 +282,8 @@ class TestClusterConductorWithSwarm(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/no_master_lb.yaml'], ['../../common/templates/environments/no_volume.yaml',
'../../common/templates/environments/no_master_lb.yaml'],
env_files) env_files)
@patch('requests.get') @patch('requests.get')
@ -345,7 +348,8 @@ class TestClusterConductorWithSwarm(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/with_master_lb.yaml'], ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/with_master_lb.yaml'],
env_files) env_files)
@patch('requests.get') @patch('requests.get')
@ -411,7 +415,8 @@ class TestClusterConductorWithSwarm(base.TestCase):
} }
self.assertEqual(expected, definition) self.assertEqual(expected, definition)
self.assertEqual( self.assertEqual(
['../../common/templates/environments/with_master_lb.yaml'], ['../../common/templates/environments/with_volume.yaml',
'../../common/templates/environments/with_master_lb.yaml'],
env_files) env_files)
@patch('magnum.conductor.utils.retrieve_cluster_template') @patch('magnum.conductor.utils.retrieve_cluster_template')

View File

@ -0,0 +1,10 @@
---
prelude: >
Currently, the swarm and the kubernetes drivers use
a dedicated cinder volume to store the container
images. It was been observed that one cinder volume
per node is a bottleneck for large clusters.
fixes:
- Make the dedicated cinder volume per node an opt-in
option. By default, no cinder volumes will be created
unless the user passes the docker-volume-size argument.