add terraform example for docker swarm

Change-Id: I249b8398fefc518c2275ae06fe283a978b51abc9
This commit is contained in:
Paul Czarkowski 2016-01-01 11:54:28 -06:00
parent 052471e03e
commit 6139e756b1
13 changed files with 546 additions and 0 deletions

4
terraform/README.md Normal file
View File

@ -0,0 +1,4 @@
# Example Terraform deployments
Contained here are example terraform deployment scripts for deploying applications and services on top of openstack.

View File

@ -0,0 +1,9 @@
*.tfvars
*.tfstate
*.backup
files/ssl/*.pem
files/ssl/*.csr
files/ssl/*.srl
templates/discovery_url

View File

@ -0,0 +1,13 @@
Copyright 2016 Paul Czarkowski
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@ -0,0 +1,216 @@
# Docker Swarm on Openstack with Terraform
Provision a Docker Swarm cluster with [Terraform](https://www.terraform.io) on Openstack.
## Status
This will install a fully HA docker swarm cluster on an Openstack Cloud. It is tested on a OpenStack Cloud provided by [BlueBox](https://www.blueboxcloud.com/) and should work on most modern installs of OpenStack that support the basic services.
It also supports overlay networks using the `docker network` command, see documentation below.
## Requirements
- [Install Terraform](https://www.terraform.io/intro/getting-started/install.html)
- Upload a CoreOS image to glance and remember the image name.
## Terraform
Terraform will be used to provision all of the OpenStack resources required to run Docker Swarm. It is also used to deploy and provision the software requirements.
### Prep
#### Openstack Authentication
Ensure your local ssh-agent is running and your ssh key has been added. This step is required by the terraform provisioner.
```
$ eval $(ssh-agent -s)
$ ssh-add ~/.ssh/id_rsa
```
Ensure that you have your Openstack credentials loaded into Terraform environment variables. Likely via a command similar to:
```
$ source ~/.stackrc
$ export TF_VAR_username=${OS_USERNAME}
$ export TF_VAR_password=${OS_PASSWORD}
$ export TF_VAR_tenant=${OS_TENANT_NAME}
$ export TF_VAR_auth_url=${OS_AUTH_URL}
```
#### General Openstack Settings
By default security_groups will allow certain traffic from `0.0.0.0/0`. If you want to restrict it to a specific network you can set the terraform variable `whitelist_network`. I like to set it to only allow my current IP:
```
$ export TF_VAR_whitelist_network=$(curl -s icanhazip.com)/32
```
You also want to specify the name of your CoreOS `glance` image as well as flavor,networks, and keys. Since these do not change often I like to add them to `terraform.tfvars`:
```
image_name = "coreos-alpha-884-0-0"
network_name = "internal"
floatingip_pool = "external"
flavor = "m1.medium"
public_key_path = "~/.ssh/id_rsa.pub"
```
_Remove the `*.tfvars` line from `.gitignore` if you wish to save this file into source control_
see `vars-openstack.tf` for the full list of variables you can set.
#### Docker Swarm Settings
You can alter the number of instances to be built and added to the cluster by modifying the `cluster_size` variable (default is 3).
If you have a FQDN you plan at pointing at one of more of the swarm-manager hosts you can set it via the `fqdn` variable.
Terraform will attempt to run `openssl` commands to create a CA and server/client certificates used to secure the docker/swarm endpoints. If you do not have `openssl` on your local machine or want to re-use existing CA / Client certificates you can set the TF variable `generate_ssl` to `0`. The certificates are created in `files/ssl`.
see `vars-swarm.tf` for the full list of variables you can set.
#### CoreOS Settings
Terraform will attempt to generate an etcd discovery token by running `curl` against the etcd discovery service. If do not have `curl` or do not wish to generate a new discovery url you can set `generate_discovery_url` to `0` and create a file `templates/discovery_url` which contains the discovery url you wish to use.
## Provision the Docker Swarm
With all your TF vars set you should be able to run `terraform apply` but lets check with `terraform plan` that things look correct first:
```
$ terraform plan
Refreshing Terraform state prior to plan...
...
...
+ template_file.discovery_url
rendered: "" => "<computed>"
template: "" => "templates/discovery_url"
Plan: 14 to add, 0 to change, 0 to destroy.
```
With no errors showing here we can go ahead and run
```
$ terraform apply
...
...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
State path: terraform.tfstate
Outputs:
swarm_cluster =
Environment Variables for accessing Docker Swarm via floating IP of first host:
export DOCKER_HOST=tcp://x.x.x.x:2375
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=/home/bacon/development/personal/terraform-dockerswarm-coreos/files/ssl
```
_the final output uses the floating IP of the first Host. You could point at any of the hosts, or use a FQDN with round robin DNS pointing at all the hosts. I avoided using neutron's load balancing service as it is not yet standard on OpenStack installs._
## Next Steps
### Check its up
copy and paste the above output into your shell and attempt to run `docker info`:
```
$ export DOCKER_HOST=tcp://x.x.x.x:2375
$ export DOCKER_TLS_VERIFY=1
$ export DOCKER_CERT_PATH=/home/bacon/development/personal/terraform-dockerswarm-coreos/files/ssl
$ docker info
Containers: 6
Images: 6
Engine Version:
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
swarm-testing-0.novalocal: 10.230.7.171:2376
└ Status: Healthy
└ Containers: 2
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 4.057 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.3.0-coreos, operatingsystem=CoreOS 884.0.0, storagedriver=overlay
swarm-testing-1.novalocal: 10.230.7.172:2376
└ Status: Healthy
└ Containers: 2
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 4.057 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.3.0-coreos, operatingsystem=CoreOS 884.0.0, storagedriver=overlay
swarm-testing-2.novalocal: 10.230.7.173:2376
└ Status: Healthy
└ Containers: 2
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 4.057 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.3.0-coreos, operatingsystem=CoreOS 884.0.0, storagedriver=overlay
CPUs: 6
Total Memory: 12.17 GiB
Name: swarm-testing-0.novalocal
```
### Create an overlay network and run a container
Create a network overlay called `my-net`
```
$ docker network create --driver overlay my-net
ecfefdff938f506b09c5ea5b505ee8ace0ee7297d9d617d06b9bbaac5bf10fea
$ docker network ls
NETWORK ID NAME DRIVER
38338f0ec63a swarm-testing-1.novalocal/host host
c41436d91f29 swarm-testing-0.novalocal/none null
e29c4451483f swarm-testing-0.novalocal/bridge bridge
400130ea105b swarm-testing-2.novalocal/none null
c8f15676b2a5 swarm-testing-2.novalocal/host host
493127ad6577 swarm-testing-2.novalocal/bridge bridge
74f862f34921 swarm-testing-1.novalocal/none null
ecfefdff938f my-net overlay
b09a38662087 swarm-testing-0.novalocal/host host
cfbcfbd7de02 swarm-testing-1.novalocal/bridge bridge
```
Run a container on the network on a specific host, then try to access it from another:
```
$ docker run -itd --name=web --net=my-net --env="constraint:node==swarm-testing-1.novalocal" nginx
53166b97adf2397403f00a2ffcdba635a7f08852c5fe4f452d6ca8c6f40bb80c
$ docker run -it --rm --net=my-net --env="constraint:node==swarm-testing-2.novalocal" busybox wget -O- http://web
Connecting to web (10.0.0.2:80)
<!DOCTYPE html>
<html>
...
...
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```
## Cleanup
Once you're done with it, don't forget to nuke the whole thing.
```
$ terraform destroy \
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
...
...
Apply complete! Resources: 0 added, 0 changed, 14 destroyed.
```

View File

@ -0,0 +1,60 @@
resource "openstack_compute_secgroup_v2" "swarm_base" {
name = "${var.cluster_name}_swarm_base"
description = "${var.cluster_name} - Docker Swarm Security Group"
# SSH
rule {
ip_protocol = "tcp"
from_port = "22"
to_port = "22"
cidr = "${var.whitelist_network}"
}
# DOCKER SWARM
rule {
ip_protocol = "tcp"
from_port = "2375"
to_port = "2375"
cidr = "${var.whitelist_network}"
}
# DOCKER
rule {
ip_protocol = "tcp"
from_port = "2376"
to_port = "2376"
cidr = "${var.whitelist_network}"
}
# INTERNAL Communication only
rule {
ip_protocol = "icmp"
from_port = "-1"
to_port = "-1"
self = true
}
rule {
ip_protocol = "tcp"
from_port = "1"
to_port = "65535"
self = true
}
rule {
ip_protocol = "udp"
from_port = "1"
to_port = "65535"
self = true
}
# DANGER DANGER DANGER
# Uncomment these if you want to allow
# unrestricted inbound access
#rule {
# ip_protocol = "tcp"
# from_port = "1"
# to_port = "65535"
# cidr = "${var.whitelist_network}"
#}
#rule {
# ip_protocol = "udp"
# from_port = "1"
# to_port = "65535"
# cidr = "${var.whitelist_network}"
#}
}

View File

@ -0,0 +1,12 @@
#!/bin/bash
openssl genrsa -out files/ssl/ca-key.pem 2048
openssl req -x509 -new -nodes -key files/ssl/ca-key.pem -days 10000 -out files/ssl/ca.pem -subj '/CN=docker-CA'
openssl genrsa -out files/ssl/key.pem 2048
openssl req -new -key files/ssl/key.pem -out files/ssl/cert.csr -subj '/CN=docker-client' -config files/ssl/openssl.cnf
openssl x509 -req -in files/ssl/cert.csr -CA files/ssl/ca.pem -CAkey files/ssl/ca-key.pem \
-CAcreateserial -out files/ssl/cert.pem -days 365 -extensions v3_req -extfile files/ssl/openssl.cnf

View File

@ -0,0 +1,8 @@
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth

View File

@ -0,0 +1,106 @@
resource "null_resource" "discovery_url_template" {
count = "${var.generate_discovery_url}"
provisioner "local-exec" {
command = "curl -s 'https://discovery.etcd.io/new?size=${var.cluster_size}' > templates/discovery_url"
}
}
resource "null_resource" "generate_ssl" {
count = "${var.generate_ssl}"
provisioner "local-exec" {
command = "bash files/ssl/generate-ssl.sh"
}
}
resource "template_file" "discovery_url" {
template = "templates/discovery_url"
depends_on = [
"null_resource.discovery_url_template"
]
}
resource "template_file" "cloud_init" {
template = "templates/cloud-init"
vars {
cluster_token = "${var.cluster_name}"
discovery_url = "${template_file.discovery_url.rendered}"
swarm_version = "${var.swarm_version}"
}
}
resource "template_file" "10_docker_service" {
template = "templates/10-docker-service.conf"
}
resource "openstack_networking_floatingip_v2" "coreos" {
count = "${var.cluster_size}"
pool = "${var.floatingip_pool}"
}
resource "openstack_compute_keypair_v2" "coreos" {
name = "swarm-${var.cluster_name}"
public_key = "${file(var.public_key_path)}"
}
resource "openstack_compute_instance_v2" "coreos" {
name = "swarm-${var.cluster_name}-${count.index}"
count = "${var.cluster_size}"
image_name = "${var.image_name}"
flavor_name = "${var.flavor}"
key_pair = "${openstack_compute_keypair_v2.coreos.name}"
network {
name = "${var.network_name}"
}
security_groups = [
"${openstack_compute_secgroup_v2.swarm_base.name}"
]
floating_ip = "${element(openstack_networking_floatingip_v2.coreos.*.address, count.index)}"
user_data = "${template_file.cloud_init.rendered}"
provisioner "file" {
source = "files"
destination = "/tmp/files"
connection {
user = "core"
}
}
provisioner "remote-exec" {
inline = [
# Create TLS certs
"mkdir -p /home/core/.docker",
"cp /tmp/files/ssl/ca.pem /home/core/.docker/",
"cp /tmp/files/ssl/cert.pem /home/core/.docker/",
"cp /tmp/files/ssl/key.pem /home/core/.docker/",
"echo 'subjectAltName = @alt_names' >> /tmp/files/ssl/openssl.cnf",
"echo '[alt_names]' >> /tmp/files/ssl/openssl.cnf",
"echo 'IP.1 = ${self.network.0.fixed_ip_v4}' >> /tmp/files/ssl/openssl.cnf",
"echo 'IP.2 = ${element(openstack_networking_floatingip_v2.coreos.*.address, count.index)}' >> /tmp/files/ssl/openssl.cnf",
"echo 'DNS.1 = ${var.fqdn}' >> /tmp/files/ssl/openssl.cnf",
"echo 'DNS.2 = ${element(openstack_networking_floatingip_v2.coreos.*.address, count.index)}.xip.io' >> /tmp/files/ssl/openssl.cnf",
"openssl req -new -key /tmp/files/ssl/key.pem -out /tmp/files/ssl/cert.csr -subj '/CN=docker-client' -config /tmp/files/ssl/openssl.cnf",
"openssl x509 -req -in /tmp/files/ssl/cert.csr -CA /tmp/files/ssl/ca.pem -CAkey /tmp/files/ssl/ca-key.pem \\",
"-CAcreateserial -out /tmp/files/ssl/cert.pem -days 365 -extensions v3_req -extfile /tmp/files/ssl/openssl.cnf",
"sudo mkdir -p /etc/docker/ssl",
"sudo cp /tmp/files/ssl/ca.pem /etc/docker/ssl/",
"sudo cp /tmp/files/ssl/cert.pem /etc/docker/ssl/",
"sudo cp /tmp/files/ssl/key.pem /etc/docker/ssl/",
# Apply localized settings to services
"sudo mkdir -p /etc/systemd/system/{docker,swarm-agent,swarm-manager}.service.d",
"cat <<'EOF' > /tmp/10-docker-service.conf\n${template_file.10_docker_service.rendered}\nEOF",
"sudo mv /tmp/10-docker-service.conf /etc/systemd/system/docker.service.d/",
"sudo systemctl daemon-reload",
"sudo systemctl restart docker.service",
"sudo systemctl start swarm-agent.service",
"sudo systemctl start swarm-manager.service",
]
connection {
user = "core"
}
}
depends_on = [
"template_file.cloud_init"
]
}
output "swarm_cluster" {
value = "\nEnvironment Variables for accessing Docker Swarm via floating IP of first host:\nexport DOCKER_HOST=tcp://${openstack_networking_floatingip_v2.coreos.0.address}:2375\nexport DOCKER_TLS_VERIFY=1\nexport DOCKER_CERT_PATH=${path.module}/files/ssl"
}

View File

@ -0,0 +1,2 @@
[Service]
Environment="DOCKER_OPTS=-H=0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert=/etc/docker/ssl/ca.pem --tlscert=/etc/docker/ssl/cert.pem --tlskey=/etc/docker/ssl/key.pem --cluster-advertise eth0:2376 --cluster-store etcd://127.0.0.1:2379/docker"

View File

@ -0,0 +1,50 @@
#cloud-config
coreos:
units:
- name: etcd.service
mask: true
- name: etcd2.service
command: start
- name: docker.service
command: start
- name: swarm-agent.service
command: enable
content: |
[Unit]
Description=swarm agent
Requires=docker.service
After=docker.service
[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=20m
ExecStartPre=/usr/bin/docker pull swarm:${swarm_version}
ExecStartPre=-/usr/bin/docker rm -f swarm-agent
ExecStart=/bin/sh -c "/usr/bin/docker run --rm --name swarm-agent swarm:${swarm_version} join --addr=$COREOS_PRIVATE_IPV4:2376 etcd://$COREOS_PRIVATE_IPV4:2379/docker"
ExecStop=/usr/bin/docker stop swarm-agent
- name: swarm-manager.service
command: enable
content: |
[Unit]
Description=swarm manager
Requires=docker.service
After=docker.service
[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=20m
ExecStartPre=/usr/bin/docker pull swarm:${swarm_version}
ExecStartPre=-/usr/bin/docker rm -f swarm-manager
ExecStart=/bin/sh -c "/usr/bin/docker run --rm --name swarm-manager -v /etc/docker/ssl:/etc/docker/ssl --net=host swarm:${swarm_version} manage --tlsverify --tlscacert=/etc/docker/ssl/ca.pem --tlscert=/etc/docker/ssl/cert.pem --tlskey=/etc/docker/ssl/key.pem etcd://$COREOS_PRIVATE_IPV4:2379/docker"
ExecStop=/usr/bin/docker stop swarm-manager
etcd2:
discovery: ${discovery_url}
advertise-client-urls: http://$private_ipv4:2379
initial-advertise-peer-urls: http://$private_ipv4:2380
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
data-dir: /var/lib/etcd2
initial-cluster-token: ${cluster_token}
update:
reboot-strategy: "off"

View File

@ -0,0 +1,4 @@
variable "generate_discovery_url" {
default = 1
description = "set to 0 if you do not want to autogenerate the discovery url"
}

View File

@ -0,0 +1,41 @@
variable "image_name" {
default = "coreos"
}
variable "network_name" {
default = "internal"
}
variable "floatingip_pool" {
default = "external"
}
variable "flavor" {
default = "m1.medium"
}
variable "username" {
description = "Your openstack username"
}
variable "password" {
description = "Your openstack password"
}
variable "tenant" {
description = "Your openstack tenant/project"
}
variable "auth_url" {
description = "Your openstack auth URL"
}
variable "public_key_path" {
description = "The path of the ssh pub key"
default = "~/.ssh/id_rsa.pub"
}
variable "whitelist_network" {
description = "network to allow connectivity from"
default = "0.0.0.0/0"
}

View File

@ -0,0 +1,21 @@
variable "cluster_size" {
default = 3
}
variable "cluster_name" {
default = "testing"
}
variable "swarm_version" {
default = "latest"
}
variable "generate_ssl" {
description = "set to 0 if you want to reuse ssl certs"
default = 1
}
variable "fqdn" {
description = "Fully Qualified DNS to add to TLS certs"
default = "swarm.example.com"
}