openstack/heat-templates
Code Issues Proposed changes

heat-config-kubelet hook

Browse Source 
This hook uses the kubelet agent from the kubernetes project
to provision containers. The StructuredConfig resource data
represents a pod of containers to be provisioned.

The files have the following purpose:
- extra-data.d/50-docker-images allows an archive file of docker images to
  be included in the dib image
- install.d/50-heat-config-kubelet installs kubernetes for redhat based distros
  during dib image build, along with the required systemd and config files required
  to enable a working kubelet service on the host
- install.d/hook-kubelet.py polls docker images and containers until the expected
  kubelet-provisioned containers are running (or a timeout occurs)
- os-refresh-config/configure.d/50-heat-config-kubelet runs before 55-heat-config
  (and the kubelet hook it triggers). This orc script writes out all pod definition files
  for the pods that should currently be running. Kubelet is configured to monitor
  the directory containing these files, so the current running containers will
  change when kubelet acts on these config changes

Author: Steve Baker <sbaker@redhat.com>
Co-Authored-By: Rabi Mishra <ramishra@redhat.com>

Change-Id: Iabb844c7d8a6f916093c364ad8caa0dbe50df3dd
This commit is contained in:
Steve Baker
2015-04-02 08:41:38 +05:30
committed by Rabi Mishra
co-authored by Rabi Mishra
parent 8bfade8908
commit 841ee38a35
11 changed files with 738 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hot/software-config/elements/README.rst
+1
View File
@@ -36,6 +36,7 @@ with the following:
heat-config-ansible \
heat-config-cfn-init \
heat-config-docker-compose \
heat-config-kubelet \
heat-config-puppet \
heat-config-salt \
heat-config-script \
hot/software-config/elements/heat-config-kubelet/README.rst
+22
View File
@@ -0,0 +1,22 @@
This hook uses the kubelet agent from the kubernetes project to provision
containers. The StructuredConfig resource data represents a pod of containers
to be provisioned.
The files have the following purpose:
- extra-data.d/50-docker-images allows an archive file of docker images to
be included in the dib image
- install.d/50-heat-config-kubelet installs kubernetes for redhat based
distros during dib image build, along with the required systemd and config
files required to enable a working kubelet service on the host
- install.d/hook-kubelet.py polls docker images and containers until the
expected kubelet-provisioned containers are running (or a timeout occurs)
- os-refresh-config/configure.d/50-heat-config-kubelet runs before
55-heat-config (and the kubelet hook it triggers). This orc script writes
out all pod definition files for the pods that should currently be running.
Kubelet is configured to monitor the directory containing these files, so
the current running containers will change when kubelet acts on these
config changes
hot/software-config/elements/heat-config-kubelet/element-deps
+2
View File
@@ -0,0 +1,2 @@
os-apply-config
os-refresh-config
hot/software-config/elements/heat-config-kubelet/extra-data.d/50-docker-images
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/bash
set -eu
set -o pipefail
if [ -z "${HEAT_DOCKER_IMAGE_ARCHIVE:-}" ]; then
echo "HEAT_DOCKER_IMAGE_ARCHIVE not set for heat-config-kubelet element" >&2
exit 0
fi
sudo mkdir -p $TMP_MOUNT_PATH/opt/heat-docker
sudo cp $HEAT_DOCKER_IMAGE_ARCHIVE $TMP_MOUNT_PATH/opt/heat-docker/images.tar
hot/software-config/elements/heat-config-kubelet/install.d/50-heat-config-kubelet
Executable
+81
View File
@@ -0,0 +1,81 @@
#!/bin/bash
set -eux
if [[ "rhel rhel7 centos7 fedora" =~ "$DISTRO_NAME" ]]; then
yum -y install --enablerepo=updates-testing kubernetes bridge-utils
cat > /etc/sysconfig/network-scripts/ifcfg-cbr0 <<EOF
DEVICE=cbr0
TYPE=Bridge
IPADDR=10.240.1.1
NETMASK=255.255.255.0
ONBOOT=yes
STP=yes
MTU=1450
# With the default forwarding delay of 15 seconds,
# many operations in a 'docker build' will simply timeout
# before the bridge starts forwarding.
DELAY=2
EOF
cat > /etc/sysconfig/network-scripts/route-cbr0 <<EOF
10.240.0.0/16 dev cbr0 scope link src 10.240.1.1
EOF
# defer docker starting until cbr0 is up
cat > /etc/systemd/system/docker.service <<EOF
.include /usr/lib/systemd/system/docker.service
[Unit]
After=network-online.target docker.socket
EOF
cat > /etc/systemd/system/heat-config-kubelet-nat-rule.service <<EOF
[Unit]
Description=iptables rule to allow nat masquerading out of 10.240.1.0/24
[Service]
ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -s 10.240.1.0/24 -j MASQUERADE
Type=oneshot
[Install]
WantedBy=multi-user.target
EOF
if [ -f "/opt/heat-docker/images.tar" ]; then
cat > /etc/systemd/system/heat-config-kubelet-load-images.service <<EOF
[Unit]
Description=Call docker load on /opt/heat-config/images.tar
After=docker.service
Before=os-collect-config.service kubelet.service
[Service]
ExecStart=/bin/docker load -i /opt/heat-docker/images.tar
ExecStart=/bin/rm -f /opt/heat-docker/images.tar
Type=oneshot
[Install]
WantedBy=multi-user.target
EOF
systemctl enable heat-config-kubelet-load-images.service
fi
cat > /etc/sysconfig/docker <<EOF
OPTIONS=--selinux-enabled --bridge cbr0 --mtu 1450 --iptables=false --insecure-registry 192.168.20.112:5001
EOF
sed -e 's|KUBELET_ARGS=""|KUBELET_ARGS="--config=/var/lib/heat-config/heat-config-kubelet/kubelet-manifests"|g' -i /etc/kubernetes/kubelet
sed -e '/KUBE_ETCD_SERVERS/ s/^#*/#/' -i /etc/kubernetes/config
systemctl disable docker.service
systemctl enable docker.service
systemctl enable kubelet.service
systemctl enable heat-config-kubelet-nat-rule.service
systemctl disable firewalld
install -D -g root -o root -m 0755 ${SCRIPTDIR}/hook-kubelet.py /var/lib/heat-config/hooks/kubelet
else
echo "Distribution '$DISTRO_NAME' is not supported"
exit 1
fi
hot/software-config/elements/heat-config-kubelet/install.d/hook-kubelet.py
Executable
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import cStringIO
import json
import logging
import os
import re
import six
import sys
import time
try:
import docker
except ImportError:
docker = None
DOCKER_BASE_URL = os.environ.get('DOCKER_HOST',
'unix:///var/run/docker.sock')
DEFAULT_IMAGES_TIMEOUT = 600
DEFAULT_CONTAINERS_TIMEOUT = 120
DEFAULT_POLL_PERIOD = 5
def get_client(log):
kwargs = {}
kwargs['base_url'] = DOCKER_BASE_URL
log.debug('Connecting to %s' % DOCKER_BASE_URL)
client = docker.Client(**kwargs)
client._version = client.version()['ApiVersion']
log.debug('Connected to version %s' % client._version)
return client
def id_to_pod_name_part(config_id):
return config_id.replace('-', '')[:15]
def container_pattern(config_id, container_name):
return '^/k8s_%s\.[0-9a-z]{8}_%s' % (
container_name, id_to_pod_name_part(config_id))
def required_images(c):
containers = c['config'].get('containers', [])
return set(container['image'] for container in containers)
def required_container_patterns(c):
config_id = c['id']
containers = c['config'].get('containers', [])
return dict((container['name'], container_pattern(
config_id, container['name'])) for container in containers)
def configure_logging():
log = logging.getLogger('heat-config')
log.setLevel('DEBUG')
formatter = logging.Formatter(
'[%(asctime)s] (%(name)s) [%(levelname)s] %(message)s')
# debug log to stderr
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(formatter)
log.addHandler(handler)
deploy_stdout = cStringIO.StringIO()
handler = logging.StreamHandler(deploy_stdout)
handler.setFormatter(formatter)
handler.setLevel('DEBUG')
log.addHandler(handler)
deploy_stderr = cStringIO.StringIO()
handler = logging.StreamHandler(deploy_stderr)
handler.setFormatter(formatter)
handler.setLevel('WARN')
log.addHandler(handler)
return log, deploy_stdout, deploy_stderr
def wait_required_images(client, log, images_timeout, poll_period, images):
log.info(
'Waiting for images: %s' % ', '.join(images))
timeout = time.time() + images_timeout
def image_prefixes(images):
for image in images:
if ':' in image:
yield image
else:
yield '%s:' % image
matching_prefixes = list(image_prefixes(images))
def image_names(all_images):
for image in all_images:
for name in image['RepoTags']:
yield name
while matching_prefixes:
all_images = list(image_names(client.images()))
for image_prefix in matching_prefixes:
for image in all_images:
if image.startswith(image_prefix):
log.info('Found image: %s' % image)
matching_prefixes.remove(image_prefix)
if time.time() > timeout:
raise Exception('Timed out after %s seconds waiting for '
'matching images: %s' % (
images_timeout,
', '.join(matching_prefixes)))
if poll_period:
time.sleep(poll_period)
def wait_required_containers(client, log,
containers_timeout, poll_period,
container_patterns):
patterns = container_patterns.values()
log.info(
'Waiting for containers matching: %s' % ', '.join(patterns))
timeout = time.time() + containers_timeout
def containers_names(containers):
for container in containers:
for name in container['Names']:
yield name
waiting_for = dict((v, re.compile(v)) for v in patterns)
while waiting_for:
for name in containers_names(client.containers()):
for k, v in six.iteritems(waiting_for):
if v.match(name):
log.info('Pattern %s matches: %s' % (k, name))
del(waiting_for[k])
break
if time.time() > timeout:
raise Exception('Timed out after %s seconds waiting for '
'matching containers: %s' % (
containers_timeout,
', '.join(waiting_for.keys)))
if poll_period:
time.sleep(poll_period)
def main(argv=sys.argv, sys_stdin=sys.stdin, sys_stdout=sys.stdout):
(log, deploy_stdout, deploy_stderr) = configure_logging()
client = get_client(log)
c = json.load(sys.stdin)
images_timeout = c['options'].get(
'images_timeout', DEFAULT_IMAGES_TIMEOUT)
containers_timeout = c['options'].get(
'containers_timeout', DEFAULT_CONTAINERS_TIMEOUT)
poll_period = c['options'].get(
'poll_period', DEFAULT_POLL_PERIOD)
pod_state = 0
try:
wait_required_images(
client,
log,
images_timeout,
poll_period,
required_images(c))
wait_required_containers(
client,
log,
containers_timeout,
poll_period,
required_container_patterns(c))
except Exception as ex:
pod_state = 1
log.error('An error occurred deploying pod %s' % c['id'])
log.exception(ex)
response = {
'deploy_stdout': deploy_stdout.getvalue(),
'deploy_stderr': deploy_stderr.getvalue(),
'deploy_status_code': pod_state,
}
json.dump(response, sys_stdout)
if __name__ == '__main__':
sys.exit(main(sys.argv))
hot/software-config/elements/heat-config-kubelet/os-refresh-config/configure.d/50-heat-config-kubelet
Executable
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import glob
import json
import logging
import os
import subprocess
import sys
import requests
MANIFESTS_DIR = os.environ.get('HEAT_KUBELET_MANIFESTS',
'/var/lib/heat-config/heat-config-kubelet'
'/kubelet-manifests')
CONF_FILE = os.environ.get('HEAT_SHELL_CONFIG',
'/var/run/heat-config/heat-config')
def main(argv=sys.argv):
log = logging.getLogger('heat-config')
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(
logging.Formatter(
'[%(asctime)s] (%(name)s) [%(levelname)s] %(message)s'))
log.addHandler(handler)
log.setLevel('DEBUG')
if not os.path.exists(CONF_FILE):
log.error('No config file %s' % CONF_FILE)
return 1
if not os.path.isdir(MANIFESTS_DIR):
os.makedirs(MANIFESTS_DIR, 0o700)
for f in glob.glob('%s/*.json'):
os.remove(f)
try:
configs = json.load(open(CONF_FILE))
except ValueError:
pass
else:
for c in configs:
try:
write_manifest(c)
except Exception as e:
log.exception(e)
def write_manifest(c):
group = c.get('group')
if group != 'kubelet':
return
fn = os.path.join(MANIFESTS_DIR, '%s.json' % c['id'])
with os.fdopen(os.open(fn, os.O_CREAT | os.O_WRONLY, 0o600), 'w') as f:
json.dump(c['config'], f, indent=2)
if __name__ == '__main__':
sys.exit(main(sys.argv))
hot/software-config/example-templates/example-kubelet-template.yaml
+75
View File
@@ -0,0 +1,75 @@
heat_template_version: 2013-05-23
parameters:
key_name:
type: string
default: heat_key
flavor:
type: string
default: m1.small
image:
type: string
default: fedora-software-config
resources:
the_sg:
type: OS::Neutron::SecurityGroup
properties:
name: the_sg
description: Ping and SSH
rules:
- protocol: icmp
- protocol: tcp
port_range_min: 22
port_range_max: 22
- protocol: tcp
port_range_min: 8080
port_range_max: 8080
kubelet_config:
type: OS::Heat::StructuredConfig
properties:
group: kubelet
options:
images_timeout: 600
containers_timeout: 120
poll_period: 10
config:
version: v1beta2
containers:
- name: simple-echo
image: busybox
command: ['nc', '-p', '8080', '-l', '-l', '-e', 'echo', 'hello world!']
ports:
- name: nc-echo
hostPort: 8080
containerPort: 8080
kubelet_deployment:
type: OS::Heat::SoftwareDeployment
properties:
name: kubelet_deployment
config:
get_resource: kubelet_config
server:
get_resource: server
server:
type: OS::Nova::Server
properties:
image: {get_param: image}
flavor: {get_param: flavor}
key_name: {get_param: key_name}
security_groups:
- {get_resource: the_sg}
user_data_format: SOFTWARE_CONFIG
outputs:
status_code_deployment:
value:
get_attr: [kubelet_deployment, deploy_status_code]
stdout:
value:
get_attr: [kubelet_deployment, deploy_stdout]
stderr:
value:
get_attr: [kubelet_deployment, deploy_stderr]
tests/software_config/hook_kubelet.py
Symlink
+1
View File
@@ -0,0 +1 @@
../../hot/software-config/elements/heat-config-kubelet/install.d/hook-kubelet.py
tests/software_config/test_heat_config_kubelet.py
+147
View File
@@ -0,0 +1,147 @@
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import os
import tempfile
import fixtures
from testtools import matchers
from tests.software_config import common
class HeatConfigKubeletORCTest(common.RunScriptTest):
fake_hooks = ['kubelet']
data = [{
"id": "abcdef001",
"group": "kubelet",
"name": "mysql",
"config": {
"version": "v1beta2",
"volumes": [{
"name": "mariadb-data"
}],
"containers": [{
"image": "mariadb_image",
"volumeMounts": [{
"mountPath": "/var/lib/mysql",
"name": "mariadb-data"
}],
"name": "mariadb",
"env": [{
"name": "DB_ROOT_PASSWORD",
"value": "mariadb_password"
}],
"ports": [{
"containerPort": 3306
}]
}]}
}, {
"id": "abcdef002",
"group": "kubelet",
"name": "rabbitmq",
"config": {
"version": "v1beta2",
"containers": [{
"image": "rabbitmq_image",
"name": "rabbitmq",
"ports": [{
"containerPort": 5672
}]
}]
}
}, {
"id": "abcdef003",
"group": "kubelet",
"name": "heat_api_engine",
"config": {
"version": "v1beta2",
"containers": [{
"image": "heat_engine_image",
"name": "heat-engine",
"env": [{
"name": "DB_ROOT_PASSWORD",
"value": "mariadb_password"
}, {
"name": "HEAT_DB_PASSWORD",
"value": "heatdb_password"
}, {
"name": "HEAT_KEYSTONE_PASSWORD",
"value": "password"
}]
}, {
"image": "heat_api_image",
"name": "heat-api",
"ports": [{
"containerPort": 8004
}]
}]
}
}]
def setUp(self):
super(HeatConfigKubeletORCTest, self).setUp()
self.fake_hook_path = self.relative_path(__file__, 'hook-fake.py')
self.heat_config_kubelet_path = self.relative_path(
__file__,
'../..',
'hot/software-config/elements',
'heat-config-kubelet/os-refresh-config/configure.d/'
'50-heat-config-kubelet')
self.manifests_dir = self.useFixture(fixtures.TempDir())
with open(self.fake_hook_path) as f:
fake_hook = f.read()
for hook in self.fake_hooks:
hook_name = self.manifests_dir.join(hook)
with open(hook_name, 'w') as f:
os.utime(hook_name, None)
f.write(fake_hook)
f.flush()
os.chmod(hook_name, 0o755)
def write_config_file(self, data):
config_file = tempfile.NamedTemporaryFile()
config_file.write(json.dumps(data))
config_file.flush()
return config_file
def test_run_heat_config(self):
with self.write_config_file(self.data) as config_file:
env = os.environ.copy()
env.update({
'HEAT_KUBELET_MANIFESTS': self.manifests_dir.join(),
'HEAT_SHELL_CONFIG': config_file.name
})
returncode, stdout, stderr = self.run_cmd(
[self.heat_config_kubelet_path], env)
self.assertEqual(0, returncode, stderr)
for config in self.data:
manifest_name = '%s.json' % config['id']
manifest_path = self.manifests_dir.join(manifest_name)
self.assertThat(manifest_path, matchers.FileExists())
#manifest file should match manifest config
self.assertEqual(config['config'],
self.json_from_file(manifest_path))
tests/software_config/test_hook_kubelet.py
+114
View File
@@ -0,0 +1,114 @@
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import mock
import re
import testtools
from tests.software_config import hook_kubelet
class HookKubeletTest(testtools.TestCase):
config = {
"id": "a50ae8dd-b0c4-407f-8732-3571b3a0f28b",
"group": "kubelet",
"inputs": [],
"name": "20_apache_deployment",
"outputs": [],
"options": {},
"config": {
"version": "v1beta2",
"volumes": [{
"name": "mariadb-data"
}],
"containers": [{
"image": "kollaglue/fedora-rdo-rabbitmq",
"name": "rabbitmq",
"ports": [{
"containerPort": 5672,
"hostPort": 5672}]
}, {
"image": "kollaglue/fedora-rdo-heat-engine",
"name": "heat-engine",
"env": [{
"name": "AUTH_ENCRYPTION_KEY",
"value": "Vegu95l2jwkucD9RSYAoFpRbUlh0PGF7"}]
}, {
"image": "kollaglue/fedora-rdo-heat-engine",
"name": "heat-engine2",
"env": [{
"name": "AUTH_ENCRYPTION_KEY",
"value": "Vegu95l2jwkucD9RSYAoFpRbUlh0PGF7"}]
}]
}
}
def setUp(self):
super(HookKubeletTest, self).setUp()
docker = mock.MagicMock()
self.docker_client = mock.MagicMock()
docker.Client.return_value = self.docker_client
self.docker_client.version.return_value = {
'ApiVersion': '1.3.0'
}
hook_kubelet.docker = docker
def test_id_to_pod_name_part(self):
self.assertEqual(
'fc9070b3ba4e4f2',
hook_kubelet.id_to_pod_name_part(
'fc9070b3-ba4e-4f22-b732-5ffdcdb40b74'))
def test_container_pattern(self):
pattern = hook_kubelet.container_pattern(
'fc9070b3-ba4e-4f22-b732-5ffdcdb40b74', 'mariadb')
self.assertEqual(
'^/k8s_mariadb\\.[0-9a-z]{8}_fc9070b3ba4e4f2', pattern)
pat = re.compile(pattern)
self.assertIsNotNone(pat.match(
'/k8s_mariadb.dac8ccce_fc9070b3ba4e4f2'
'uv6pejpu5nqbmrqoungurhtob5gvt.default.'
'file_2c8cf9fc94674e8buv6pejpu5nqbmrqoungurhtob5gvt_dcd1e1d9'))
self.assertIsNotNone(pat.match(
'/k8s_mariadb.dac8ccce_fc9070b3ba4e4f2a'))
self.assertIsNone(pat.match(
'k8s_mariadb.dac8ccce_fc9070b3ba4e4f2a'))
self.assertIsNone(pat.match(
'/k8s_mysqldb.dac8ccce_fc9070b3ba4e4f2a'))
self.assertIsNone(pat.match(
'/k8s_mariadb.dac8ccc_fc9070b3ba4e4f2a'))
self.assertIsNone(pat.match(
'/k8s_mariadb.dac8ccce_gc9070b3ba4e4f22a'))
def test_required_images(self):
self.assertEqual(
set([
'kollaglue/fedora-rdo-heat-engine',
'kollaglue/fedora-rdo-rabbitmq']),
hook_kubelet.required_images(self.config))
self.assertEqual(
set(), hook_kubelet.required_images({'config': {}}))
def test_required_container_patterns(self):
patterns = hook_kubelet.required_container_patterns(self.config)
self.assertEqual({
'heat-engine': '^/k8s_heat-engine\\.[0-9a-z]{8}_a50ae8ddb0c4407',
'heat-engine2': '^/k8s_heat-engine2\\.[0-9a-z]{8}_a50ae8ddb0c4407',
'rabbitmq': '^/k8s_rabbitmq\\.[0-9a-z]{8}_a50ae8ddb0c4407'
}, patterns)