From e136f3602852542062ad4971372666469efa0f37 Mon Sep 17 00:00:00 2001 From: Ton Ngo Date: Tue, 17 May 2016 15:45:28 +0000 Subject: [PATCH] Add Storage section in user guide Describe how to use Cinder block with COE's This section covers both ephemeral and persistent storage for containers. It describes how Cinder block is used by Kubernetes, Swarm, Mesos, and how Magnum is providing support along with the options for users to manage the storage. Examples are included. The doc container-volume-integration.rst is updated to the current status and incorporated into the user guide. Partially implements: blueprint user-guide Change-Id: Iecd1a58e7bc04dda905e4c43d709f9eec7f68b97 --- .../dev/container-volume-integration.rst | 235 -------------- doc/source/index.rst | 1 - doc/source/userguide.rst | 286 ++++++++++++++++++ 3 files changed, 286 insertions(+), 236 deletions(-) delete mode 100644 doc/source/dev/container-volume-integration.rst diff --git a/doc/source/dev/container-volume-integration.rst b/doc/source/dev/container-volume-integration.rst deleted file mode 100644 index 81c2c6b62d..0000000000 --- a/doc/source/dev/container-volume-integration.rst +++ /dev/null @@ -1,235 +0,0 @@ -=============================================== -Using Container Volume Integration Feature -=============================================== - -For Magnum user, we use Container Volume Integration in Kubernetes, Swarm -and Mesos. This document details instructions how to use the container volume -integration in Kubernetes, Swarm and Mesos. - -Using container volume integration in Kubernetes ------------------------------------------------- - -**NOTE:** The Container Volume Model of Kubernetes needs the kubernetes -version >= 1.1.1 and docker version >= 1.8.3 - -1. Store the Fedora Atomic micro-OS in glance - - The steps for updating Fedora Atomic images are a bit detailed.Fortunately - one of the core developers has made Atomic images available at - https://fedorapeople.org/groups/magnum:: - - cd ~ - wget https://fedorapeople.org/groups/magnum/fedora-23-atomic-7.qcow2 - glance image-create --name fedora-23-atomic-7 \ - --visibility public \ - --disk-format qcow2 \ - --os-distro fedora-atomic \ - --container-format bare < fedora-23-atomic-7.qcow2 - -2. Edit the file:: - - sudo vi /opt/stack/magnum/magnum/templates/kubernetes/kubecluster.yaml - - The default value of the kube_version entries needs to change 'v1.0.6' to - 'v1.1.8', then restart magnum-conduct service.(Magnum team will make the - step automation as soon as possible.) - -3. Create the baymodel. - - The new attribute volume-driver for a baymodel specifies the volume backend - driver to use when deploying a bay. The volume-driver value needs to be - specified as 'cinder' for kubernetes:: - - magnum baymodel-create --name k8sbaymodel \ - --image-id fedora-23-atomic-7 \ - --keypair-id testkey \ - --external-network-id public \ - --dns-nameserver 8.8.8.8 \ - --flavor-id m1.small \ - --docker-volume-size 5 \ - --network-driver flannel \ - --coe kubernetes \ - --volume-driver cinder - -4. Create the bay:: - - magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1 - -To enable the container volume integration in kubernetes, log into each minion -node of your bay and perform the following step 5, step 6, step 7 and step 8: - -5. Configure kubelet:: - - sudo vi /etc/kubernetes/kubelet - - Comment out the line:: - - #KUBELET_ARGS=--config=/etc/kubernetes/manifests --cadvisor-port=4194 - - Uncomment the line:: - - #KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config" - -**NOTE:** This is a temporary workaround, and Magnum team is working -on a long term solution to automate this step. - -6. Enter OpenStack user credential:: - - sudo vi /etc/kubernetes/kube_openstack_config - - The username, tenant-name and region entries have been filled in with the - Keystone values of the user who created the bay. Enter the password - of this user on the entry for password:: - - password=ChangeMe - -7. Restart Kubernetes services:: - - sudo systemctl restart kubelet - -8. Run the docker container:: - - sudo docker run -v /usr/local/bin:/target jpetazzo/nsenter - -Kubernetes container volume integration has configured by the above steps, -And you can use the kubernetes container volume now. The following steps is -an example for container volume integration with kubernetes bay. - -1. Create the cinder volume:: - - cinder create --display-name=test-repo 1 - - ID=$(cinder create --display-name=test-repo 1 | awk -F'|' '$2~/^[[:space:]]*id/ {print $3}') - - The command will generate the volume with a ID. The volume ID will be specified in - Step 2. - -2. Create a container in this bay. - -The following example illustrates how to mount an cinder volume for a pod. - -Create a file (e.g nginx-cinder.yaml) describing a pod:: - - cat > nginx-cinder.yaml << END - apiVersion: v1 - kind: Pod - metadata: - name: aws-web - spec: - containers: - - name: web - image: nginx - ports: - - name: web - containerPort: 80 - hostPort: 8081 - protocol: TCP - volumeMounts: - - name: html-volume - mountPath: "/usr/share/nginx/html" - volumes: - - name: html-volume - cinder: - # Enter the volume ID below - volumeID: $ID - fsType: ext4 - END - -**NOTE:** The cinder volume ID needs to be configured into the yaml file -so that an existing Cinder volume can be mounted in a pod by specifying -the volume ID in the pod manifest as follows:: - - volumes: - - name: html-volume - cinder: - volumeID: $ID - fsType: ext4 - -3. Create a pod with container. Please refer to the quickstart guide on how to - connect to Kubernetes running on the launched bay.:: - - kubectl create -f ./nginx-cinder.yaml - -You can log in the container to check if existing the mountPath, and check -if your cinder volume status is 'in-use' by running the command 'cinder list'. - -Using container volume integration in Swarm -------------------------------------------- -*To be filled in* - -Using container volume integration in Mesos -------------------------------------------- - -1. Create the baymodel. - - One of the new attributes volume-driver for a baymodel specifies the volume - backend driver to use when deploying a bay. The volume-driver value needs to - be specified as rexray for Mesos. - The other new attributes rexray_preempt for a baymodel is an optional - parameter here which enables any host to take control of a volume - irrespective of whether other hosts are using the volume. If this is set to - false then mostly plugins ensure safety first for locking the volume:: - - magnum baymodel-create --name mesosbaymodel \ - --image-id ubuntu-mesos \ - --keypair-id testkey \ - --external-network-id public \ - --dns-nameserver 8.8.8.8 \ - --master-flavor-id m1.magnum \ - --docker-volume-size 4 \ - --tls-disabled \ - --flavor-id m1.magnum \ - --coe mesos \ - --volume-driver rexray \ - --labels rexray-preempt=true - -2. Create the mesos bay:: - - magnum bay-create --name mesosbay --baymodel mesosbaymodel --node-count 1 - -3. Create the cinder volume and a container in this bay:: - - cinder create --display-name=redisdata 1 - - Create the following mesos.json file:: - - cat > mesos.json << END - { - "id": "redis", - "container": { - "docker": { - "image": "redis", - "network": "BRIDGE", - "portMappings": [ - { "containerPort": 80, "hostPort": 0, "protocol": "tcp"} - ], - "parameters": [ - { "key": "volume-driver", "value": "rexray" }, - { "key": "volume", "value": "redisdata:/data" } - ] - } - }, - "cpus": 0.2, - "mem": 32.0, - "instances": 1 - } - END - -**NOTE:** When the mesos bay is created using this baymodel, the mesos bay -will be configured so that an existing cinder volume can be mounted in a -container by configuring the parameters to mount the cinder volume in the -json file:: - - "parameters": [ - { "key": "volume-driver", "value": "rexray" }, - { "key": "volume", "value": "redisdata:/data" } - ] - -4. Using the REST API of Marathon:: - - MASTER_IP=$(magnum bay-show mesosbay | awk '/ api_address /{print $4}') - curl -X POST -H "Content-Type: application/json" \ - http://${MASTER_IP}:8080/v2/apps -d@mesos.json - -You can log in the container to check if existing the mountPath, and check -if your cinder volume status is 'in-use' by running the command 'cinder list'. diff --git a/doc/source/index.rst b/doc/source/index.rst index 9e2eb5b68e..df5b3b4c30 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -80,7 +80,6 @@ Developer Info dev/bay-template-example.rst dev/build-atomic-image.rst dev/kubernetes-load-balancer.rst - dev/container-volume-integration.rst dev/tls.rst dev/mesos.rst dev/functional-test.rst diff --git a/doc/source/userguide.rst b/doc/source/userguide.rst index bb0c6e810e..16ee766ff9 100644 --- a/doc/source/userguide.rst +++ b/doc/source/userguide.rst @@ -309,8 +309,294 @@ Include Autoscaling ======= Storage ======= + +Currently Cinder provides the block storage to the containers, and the +storage is made available in two ways: as ephemeral storage and as +persistent storage. + +Ephemeral storage +----------------- + +The filesystem for the container consists of multiple layers from the +image and a top layer that holds the modification made by the +container. This top layer requires storage space and the storage is +configured in the Docker daemon through a number of storage options. +When the container is removed, the storage allocated to the particular +container is also deleted. + +To manage this space in a flexible manner independent of the Nova +instance flavor, Magnum creates a separate Cinder block volume for each +node in the bay, mounts it to the node and configures it to be used as +ephemeral storage. Users can specify the size of the Cinder volume with +the baymodel attribute 'docker-volume-size'. The default size is 5GB. +Currently the block size is fixed at bay creation time, but future +lifecycle operations may allow modifying the block size during the +life of the bay. + +To use the Cinder block storage, there is a number of Docker +storage drivers available. Only 'devicemapper' is supported as the +storage driver but other drivers such as 'OverlayFS' are being +considered. There are important trade-off between the choices +for the storage drivers that should be considered. For instance, +'OperlayFS' may offer better performance, but it may not support +the filesystem metadata needed to use SELinux, which is required +to support strong isolation between containers running in the same +bay. Using the 'devicemapper' driver does allow the use of SELinux. + + +Persistent storage +------------------ + +In some use cases, data read/written by a container needs to persist +so that it can be accessed later. To persist the data, a Cinder +volume with a filesystem on it can be mounted on a host and be made +available to the container, then be unmounted when the container exits. + +Docker provides the 'volume' feature for this purpose: the user +invokes the 'volume create' command, specifying a particular volume +driver to perform the actual work. Then this volume can be mounted +when a container is created. A number of third-party volume drivers +support OpenStack Cinder as the backend, for example Rexray and +Flocker. Magnum currently supports Rexray as the volume driver for +Swarm and Mesos. Other drivers are being considered. + +Kubernetes allows a previously created Cinder block to be mounted to +a pod and this is done by specifying the block ID in the pod yaml file. +When the pod is scheduled on a node, Kubernetes will interface with +Cinder to request the volume to be mounted on this node, then +Kubernetes will launch the Docker container with the proper options to +make the filesystem on the Cinder volume accessible to the container +in the pod. When the pod exits, Kubernetes will again send a request +to Cinder to unmount the volume's filesystem, making it avaiable to be +mounted on other nodes. + +Magnum supports these features to use Cinder as persistent storage +using the baymodel attribute 'volume-driver' and the support matrix +for the COE types is summarized as follows: + ++--------+-------------+-------------+-------------+ +| Driver | Kubernetes | Swarm | Mesos | ++========+=============+=============+=============+ +| cinder | supported | unsupported | unsupported | ++--------+-------------+-------------+-------------+ +| rexray | unsupported | supported | supported | ++--------+-------------+-------------+-------------+ + +Following are some examples for using Cinder as persistent storage. + +Using Cinder in Kubernetes +++++++++++++++++++++++++++ + +**NOTE:** This feature requires Kubernetes version 1.1.1 or above and +Docker version 1.8.3 or above. The public Fedora image from Atomic +currently meets this requirement. + +**NOTE:** The following steps are a temporary workaround, and Magnum's +development team is working on a long term solution to automate these steps. + +1. Create the baymodel. + + Specify 'cinder' as the volume-driver for Kubernetes:: + + magnum baymodel-create --name k8sbaymodel \ + --image-id fedora-23-atomic-7 \ + --keypair-id testkey \ + --external-network-id public \ + --dns-nameserver 8.8.8.8 \ + --flavor-id m1.small \ + --docker-volume-size 5 \ + --network-driver flannel \ + --coe kubernetes \ + --volume-driver cinder + +2. Create the bay:: + + magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1 + + +3. Configure kubelet. + + To allow Kubernetes to interface with Cinder, log into each minion + node of your bay and perform step 4 through 6:: + + sudo vi /etc/kubernetes/kubelet + + Comment out the line:: + + #KUBELET_ARGS=--config=/etc/kubernetes/manifests --cadvisor-port=4194 + + Uncomment the line:: + + #KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config" + + +4. Enter OpenStack user credential:: + + sudo vi /etc/kubernetes/kube_openstack_config + + The username, tenant-name and region entries have been filled in with the + Keystone values of the user who created the bay. Enter the password + of this user on the entry for password:: + + password=ChangeMe + +5. Restart Kubernetes services:: + + sudo systemctl restart kubelet + + On restart, the new configuration enables the Kubernetes cloud provider + plugin for OpenStack, along with the necessary credential for kubelet + to authenticate with Keystone and to make request to OpenStack services. + +6. Install nsenter:: + + sudo docker run -v /usr/local/bin:/target jpetazzo/nsenter + + The nsenter utility is used by Kubernetes to run new processes within + existing kernel namespaces. This allows the kubelet agent to manage storage + for pods. + +Kubernetes is now ready to use Cinder for persistent storage. +Following is an example illustrating how Cinder is used in a pod. + +1. Create the cinder volume:: + + cinder create --display-name=test-repo 1 + + ID=$(cinder create --display-name=test-repo 1 | awk -F'|' '$2~/^[[:space:]]*id/ {print $3}') + + The command will generate the volume with a ID. The volume ID will be specified in + Step 2. + +2. Create a pod in this bay and mount this cinder volume to the pod. + Create a file (e.g nginx-cinder.yaml) describing the pod:: + + cat > nginx-cinder.yaml << END + apiVersion: v1 + kind: Pod + metadata: + name: aws-web + spec: + containers: + - name: web + image: nginx + ports: + - name: web + containerPort: 80 + hostPort: 8081 + protocol: TCP + volumeMounts: + - name: html-volume + mountPath: "/usr/share/nginx/html" + volumes: + - name: html-volume + cinder: + # Enter the volume ID below + volumeID: $ID + fsType: ext4 + END + +**NOTE:** The Cinder volume ID needs to be configured in the yaml file +so the existing Cinder volume can be mounted in a pod by specifying +the volume ID in the pod manifest as follows:: + + volumes: + - name: html-volume + cinder: + volumeID: $ID + fsType: ext4 + +3. Create the pod by the normal Kubernetes interface:: + + kubectl create -f nginx-cinder.yaml + +You can start a shell in the container to check that the mountPath exists, +and on an OpenStack client you can run the command 'cinder list' to verify +that the cinder volume status is 'in-use'. + + +Using Cinder in Swarm ++++++++++++++++++++++ *To be filled in* + +Using Cinder in Mesos ++++++++++++++++++++++ + +1. Create the baymodel. + + Specify 'rexray' as the volume-driver for Mesos. As an option, you + can specify in a label the attributes 'rexray_preempt' to enable + any host to take control of a volume regardless if other + hosts are using the volume. If this is set to false, the driver + will ensure data safety by locking the volume:: + + magnum baymodel-create --name mesosbaymodel \ + --image-id ubuntu-mesos \ + --keypair-id testkey \ + --external-network-id public \ + --dns-nameserver 8.8.8.8 \ + --master-flavor-id m1.magnum \ + --docker-volume-size 4 \ + --tls-disabled \ + --flavor-id m1.magnum \ + --coe mesos \ + --volume-driver rexray \ + --labels rexray-preempt=true + +2. Create the Mesos bay:: + + magnum bay-create --name mesosbay --baymodel mesosbaymodel --node-count 1 + +3. Create the cinder volume and configure this bay:: + + cinder create --display-name=redisdata 1 + + Create the following file :: + + cat > mesos.json << END + { + "id": "redis", + "container": { + "docker": { + "image": "redis", + "network": "BRIDGE", + "portMappings": [ + { "containerPort": 80, "hostPort": 0, "protocol": "tcp"} + ], + "parameters": [ + { "key": "volume-driver", "value": "rexray" }, + { "key": "volume", "value": "redisdata:/data" } + ] + } + }, + "cpus": 0.2, + "mem": 32.0, + "instances": 1 + } + END + +**NOTE:** When the Mesos bay is created using this baymodel, the Mesos bay +will be configured so that a filesystem on an existing cinder volume can +be mounted in a container by configuring the parameters to mount the cinder +volume in the json file :: + + "parameters": [ + { "key": "volume-driver", "value": "rexray" }, + { "key": "volume", "value": "redisdata:/data" } + ] + +4. Create the container using Marathon REST API :: + + MASTER_IP=$(magnum bay-show mesosbay | awk '/ api_address /{print $4}') + curl -X POST -H "Content-Type: application/json" \ + http://${MASTER_IP}:8080/v2/apps -d@mesos.json + +You can log into the container to check that the mountPath exists, and +you can run the command 'cinder list' to verify that your cinder +volume status is 'in-use'. + + ================ Image Management ================