airship/promenade
Code Issues Proposed changes

Merge pull request #28 from mark-burnett/generate-shell-script

Browse Source 
Drive all configuration through yaml documents
This commit is contained in:
Scott Hussey
2017-07-12 12:53:33 -05:00
committed by GitHub
parent 3900378a54 1fb686d1a1
commit 0dfcd9c5c9
11 changed files with 84 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
docs/getting-started.md
View File

@@ -7,11 +7,11 @@
Make sure you have [Vagrant](https://vagrantup.com) and
[VirtualBox](https://www.virtualbox.org/wiki/Downloads) installed.
Generate the certificates and keys to be used:
Generate the per-host configuration, certificates and keys to be used:
```bash
mkdir configs
docker run --rm -t -v $(pwd):/target quay.io/attcomdev/promenade:experimental promenade -v generate -c /target/example/vagrant-input-config.yaml -o /target/configs
docker run --rm -t -v $(pwd):/target quay.io/attcomdev/promenade:latest promenade -v generate -c /target/example/vagrant-input-config.yaml -o /target/configs
```
Start the VMs:
@@ -23,44 +23,45 @@ vagrant up
Start the genesis node:
```bash
vagrant ssh n0 -c 'sudo /vagrant/genesis.sh /vagrant/configs/n0.yaml'
vagrant ssh n0 -c 'sudo /vagrant/configs/up.sh /vagrant/configs/n0.yaml'
```
Join the master nodes:
```bash
vagrant ssh n1 -c 'sudo /vagrant/join.sh /vagrant/configs/n1.yaml'
vagrant ssh n2 -c 'sudo /vagrant/join.sh /vagrant/configs/n2.yaml'
vagrant ssh n1 -c 'sudo /vagrant/configs/up.sh /vagrant/configs/n1.yaml'
vagrant ssh n2 -c 'sudo /vagrant/configs/up.sh /vagrant/configs/n2.yaml'
```
Join the worker node:
```bash
vagrant ssh n3 -c 'sudo /vagrant/join.sh /vagrant/configs/n3.yaml'
vagrant ssh n3 -c 'sudo /vagrant/configs/up.sh /vagrant/configs/n3.yaml'
```
### Development Cleanup
To use Promenade from behind a proxy, simply add proxy settings to the
promenade `Network` configuration document using the keys `http_proxy`,
`https_proxy`, and `no_proxy` before running `generate`.
If you are testing/developing on hosts that cannot be easily destroyed, you may
find the `cleanup.sh` script useful.
Note that it is important to specify `no_proxy` to include `kubernetes` and the
IP addresses of all the master nodes.
### Building the image
```bash
docker build -t quay.io/attcomdev/promenade:experimental .
docker build -t promenade:local .
```
For development, you may wish to save it and have the `genesis.sh` and
`join.sh` scripts load it:
For development, you may wish to save it and have the `up.sh` script load it:
```bash
docker save -o promenade.tar quay.io/attcomdev/promenade:experimental
docker save -o promenade.tar promenade:local
```
Then on a node:
```bash
PROMENADE_LOAD_IMAGE=/vagrant/promenade.tar /vagrant/genesis.sh /vagrant/path/to/node-config.yaml
PROMENADE_LOAD_IMAGE=/vagrant/promenade.tar /vagrant/up.sh /vagrant/path/to/node-config.yaml
```
To build the image from behind a proxy, you can:
@@ -68,7 +69,7 @@ To build the image from behind a proxy, you can:
```bash
export http_proxy=...
export no_proxy=...
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$http_proxy --build-arg no_proxy=$no_proxy -t quay.io/attcomdev/promenade:experimental .
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$http_proxy --build-arg no_proxy=$no_proxy -t promenade:local .
```
## Using Promenade Behind a Proxy
@@ -81,5 +82,5 @@ cd /vagrant
export DOCKER_HTTP_PROXY="http://proxy.server.com:8080"
export DOCKER_HTTPS_PROXY="https://proxy.server.com:8080"
export DOCKER_NO_PROXY="localhost,127.0.0.1"
sudo -E /vagrant/genesis.sh /vagrant/configs/n0.yaml
sudo -E /vagrant/up.sh /vagrant/configs/n0.yaml
```

12
example/vagrant-input-config.yaml
View File

@@ -50,3 +50,15 @@ spec:
- 8.8.4.4
#http_proxy: http://proxy.example.com:8080
#https_proxy: https://proxy.example.com:8080
---
apiVersion: promenade/v1
kind: Versions
metadata:
cluster: example
name: example
target: all
spec:
images:
promenade: quay.io/attcomdev/promenade:latest
packages:
docker: docker.io=1.12.6-0ubuntu1~16.04.1

75
join.sh
View File

@@ -1,75 +0,0 @@
#!/usr/bin/env bash
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root." 1>&2
exit 1
fi
set -ex
#Promenade Variables
DOCKER_PACKAGE="docker.io"
DOCKER_VERSION=1.12.6-0ubuntu1~16.04.1
#Proxy Variables
DOCKER_HTTP_PROXY=${DOCKER_HTTP_PROXY:-${HTTP_PROXY:-${http_proxy}}}
DOCKER_HTTPS_PROXY=${DOCKER_HTTPS_PROXY:-${HTTPS_PROXY:-${https_proxy}}}
DOCKER_NO_PROXY=${DOCKER_NO_PROXY:-${NO_PROXY:-${no_proxy}}}
mkdir -p /etc/docker
cat <<EOS > /etc/docker/daemon.json
{
"live-restore": true,
"storage-driver": "overlay2"
}
EOS
#Configuration for Docker Behind a Proxy
mkdir -p /etc/systemd/system/docker.service.d
#Set HTTPS Proxy Variable
cat <<EOF > /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=${DOCKER_HTTP_PROXY}"
EOF
#Set HTTPS Proxy Variable
cat <<EOF > /etc/systemd/system/docker.service.d/https-proxy.conf
[Service]
Environment="HTTPS_PROXY=${DOCKER_HTTPS_PROXY}"
EOF
#Set No Proxy Variable
cat <<EOF > /etc/systemd/system/docker.service.d/no-proxy.conf
[Service]
Environment="NO_PROXY=${DOCKER_NO_PROXY}"
EOF
#Reload systemd and docker if present
systemctl daemon-reload
systemctl restart docker || true
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq --no-install-recommends \
$DOCKER_PACKAGE=$DOCKER_VERSION \
if [ -f "${PROMENADE_LOAD_IMAGE}" ]; then
echo === Loading updated promenade image ===
docker load -i "${PROMENADE_LOAD_IMAGE}"
fi
docker pull quay.io/attcomdev/promenade:experimental
docker run -t --rm \
-v /:/target \
quay.io/attcomdev/promenade:experimental \
promenade \
-v \
join \
--hostname $(hostname) \
--config-path /target$(realpath $1)
touch /var/lib/prom.done

25
promenade/cli.py
View File

@@ -25,34 +25,13 @@ def promenade(*, verbose):
type=click.Path(exists=True, file_okay=False,
dir_okay=True, resolve_path=True),
help='Location where templated files will be placed.')
def genesis(*, asset_dir, config_path, hostname, target_dir):
def up(*, asset_dir, config_path, hostname, target_dir):
op = operator.Operator.from_config(config_path=config_path,
hostname=hostname,
target_dir=target_dir)
op.genesis(asset_dir=asset_dir)
@promenade.command(help='Join an existing cluster')
@click.option('-a', '--asset-dir', default='/assets',
type=click.Path(exists=True, file_okay=False,
dir_okay=True, resolve_path=True),
help='Source path for binaries to deploy.')
@click.option('-c', '--config-path', type=click.File(),
help='Location of cluster configuration data.')
@click.option('--hostname', help='Current hostname.')
@click.option('-t', '--target-dir', default='/target',
type=click.Path(exists=True, file_okay=False,
dir_okay=True, resolve_path=True),
help='Location where templated files will be placed.')
def join(*, asset_dir, config_path, hostname, target_dir):
op = operator.Operator.from_config(config_path=config_path,
hostname=hostname,
target_dir=target_dir)
op.join(asset_dir=asset_dir)
op.up(asset_dir=asset_dir)
@promenade.command(help='Generate certs and keys')

4
promenade/config.py
View File

@@ -33,6 +33,7 @@ class Document:
'Node',
'PrivateKey',
'PublicKey',
'Versions',
}
def __init__(self, data):
@@ -68,6 +69,9 @@ class Document:
def __getitem__(self, key):
return self.data['spec'][key]
def get(self, key, default=None):
return self.data['spec'].get(key, default)
class Configuration:
def __init__(self, documents):

14
promenade/generator.py
View File

@@ -1,4 +1,4 @@
from . import config, logging, pki
from . import config, logging, pki, renderer
import os
__all__ = ['Generator']
@@ -18,7 +18,7 @@ class Generator:
self.validate()
def validate(self):
required_kinds = ['Cluster', 'Network']
required_kinds = ['Cluster', 'Network', 'Versions']
for required_kind in required_kinds:
try:
self.input_config[required_kind]
@@ -30,9 +30,17 @@ class Generator:
assert self.input_config['Cluster'].metadata['name'] \
== self.input_config['Network'].metadata['cluster']
def generate_up_sh(self, output_dir):
r = renderer.Renderer(config=self.input_config,
target_dir=output_dir)
r.render_generate_files()
def generate_all(self, output_dir):
self.generate_up_sh(output_dir)
cluster = self.input_config['Cluster']
network = self.input_config['Network']
versions = self.input_config['Versions']
cluster_name = cluster.metadata['name']
LOG.info('Generating configuration for cluster "%s"', cluster_name)
@@ -91,6 +99,7 @@ class Generator:
network,
sa_pub,
sa_priv,
versions,
]
for hostname, data in cluster['nodes'].items():
@@ -142,6 +151,7 @@ class Generator:
node,
proxy_cert,
proxy_cert_key,
versions,
]
role_specific_documents = []

8
promenade/operator.py
View File

@@ -19,13 +19,7 @@ class Operator:
self.hostname = hostname
self.target_dir = target_dir
def genesis(self, *, asset_dir=None):
self.setup(asset_dir=asset_dir)
def join(self, *, asset_dir=None):
self.setup(asset_dir=asset_dir)
def setup(self, *, asset_dir):
def up(self, *, asset_dir):
self.rsync_from(asset_dir)
self.render()

7
promenade/renderer.py
View File

@@ -15,6 +15,9 @@ class Renderer:
self.config = config
self.target_dir = target_dir
def render_generate_files(self):
self.render_template_dir('generate')
def render(self):
for template_dir in self.config['Node']['templates']:
self.render_template_dir(template_dir)
@@ -38,7 +41,9 @@ class Renderer:
LOG.debug('Templating "%s" into "%s"', path, target_path)
env = jinja2.Environment(undefined=jinja2.StrictUndefined)
env = jinja2.Environment(
loader=jinja2.PackageLoader('promenade', 'templates/include'),
undefined=jinja2.StrictUndefined)
env.filters['b64enc'] = _base64_encode
with open(path) as f:

1
promenade/templates/generate/up.sh Normal file
View File

@@ -0,0 +1 @@
{% include "up.sh" with context %}

18
promenade/templates/genesis/etc/kubernetes/asset-loader/assets/configmap-promenade-join.yaml Normal file
View File

@@ -0,0 +1,18 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: drydock
spec: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: promenade-join-sh
namespace: drydock
data:
join.sh: |-
{%- filter indent(4, True) %}
{% include "up.sh" %}
{%- endfilter %}

20
genesis.sh → promenade/templates/include/up.sh Executable file → Normal file
View File

@@ -5,14 +5,18 @@ if [ "$(id -u)" != "0" ]; then
exit 1
fi
if [ "x$1" = "x" ]; then
echo "Path to node configuration required." 1>&2
exit 1
fi
set -ex
#Promenade Variables
DOCKER_PACKAGE="docker.io"
DOCKER_VERSION=1.12.6-0ubuntu1~16.04.1
#Proxy Variables
http_proxy={{ config['Network'].get('http_proxy', '') }}
https_proxy={{ config['Network'].get('https_proxy', '') }}
no_proxy={{ config['Network'].get('no_proxy', '') }}
DOCKER_HTTP_PROXY=${DOCKER_HTTP_PROXY:-${HTTP_PROXY:-${http_proxy}}}
DOCKER_HTTPS_PROXY=${DOCKER_HTTPS_PROXY:-${HTTPS_PROXY:-${https_proxy}}}
DOCKER_NO_PROXY=${DOCKER_NO_PROXY:-${NO_PROXY:-${no_proxy}}}
@@ -54,22 +58,20 @@ systemctl restart docker || true
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq --no-install-recommends \
$DOCKER_PACKAGE=$DOCKER_VERSION \
{{ config['Versions']['packages']['docker'] }}
if [ -f "${PROMENADE_LOAD_IMAGE}" ]; then
echo === Loading updated promenade image ===
docker load -i "${PROMENADE_LOAD_IMAGE}"
fi
docker pull quay.io/attcomdev/promenade:experimental
docker run -t --rm \
--net host \
-v /:/target \
quay.io/attcomdev/promenade:experimental \
{{ config['Versions']['images']['promenade'] }} \
promenade \
-v \
genesis \
up \
--hostname $(hostname) \
--config-path /target$(realpath $1) 2>&1