openstack/kolla
Code Issues Proposed changes

Implements: blueprint heat-template

Browse Source 
Previously, the kolla heat template was maintained outside of
the project in the larsks/heat-kubernetes repo. This patch adds
the templates to the project's devenv directory. Additionally,
the patch updates the templates and documentation to reflect
the design changes outlined in the following spec:

https://github.com/stackforge/kolla/blob/master/specs/containerize-openstack.rst

Change-Id: I11305f87cceb2495478d458a7e381d20838e664a
This commit is contained in:
Daneyon Hansen 2015-02-16 21:37:03 +00:00
parent ddc12789bc
commit e0de6af3ec
6 changed files with 503 additions and 169 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

218
README.md
View File

@ -1,30 +1,36 @@
Kolla
=====
Kolla Overview
==============
The Kolla project is part of the OpenStack TripleO effort, focused on
deploying OpenStack environments using [Kubernetes][] and [Docker][]
containers.
The Kolla project is part of the OpenStack [TripleO][] effort, focused
on deploying OpenStack services using [Docker][] containers. The initial
system [spec][] provides additional details of Kolla and the use cases
it addresses.
[kubernetes]: https://github.com/GoogleCloudPlatform/kubernetes
[docker]: http://docker.com/
[TripleO]: https://wiki.openstack.org/wiki/TripleO
[Docker]: http://docker.com/
[spec]: https://github.com/stackforge/kolla/blob/master/specs/containerize-openstack.rst
Getting Started
===============
Kubernetes deployment on bare metal is a complex topic which is beyond the
scope of this project at this time. The developers require a development test
environment. As a result, one of the developers has created a Heat based
deployment tool that can be
found [here](https://github.com/larsks/heat-kubernetes).
Deployment on bare metal is a complex topic which is beyond the scope of
the project at this time. An environment to simplify the deployment of a
single or multi-node Kolla cluster is required for development purposes.
As a result, a [Heat template][] has been created for deploying a Kolla
cluster to an existing OpenStack cloud.
[Heat template]: (https://github.com/stackforge/kolla/blob/master/devenv/README.md)
Build Docker Images
-------------------
Docker Images
-------------
Images are built by the Kolla project maintainers. It is possible to build
unique images with specific changes, but these would end up in a personal
namespace. Read the docs directory image workflow documentation for more
details.
The [Docker images][] are built by the Kolla project maintainers. A detailed
process for contributing to the images can be found [here][]. Images reside
in the Docker Hub [Kollaglue repo][].
[here]: https://github.com/stackforge/kolla/blob/master/docs/image-building.md
[Docker images]: https://docs.docker.com/userguide/dockerimages/
[Kollaglue repo]: https://registry.hub.docker.com/repos/kollaglue/
The Kolla developers build images in the kollaglue namespace for the following
services:
@ -45,157 +51,6 @@ $ sudo docker search kollaglue
```
A list of the upstream built docker images will be shown.
Use Kubernetes to Deploy OpenStack
----------------------------------
At this point, we believe the key features for a minimum viable feature set
are implemented. This includes the capability to launch virtual machines in
Nova. One key fact is that networking may not entirely work properly yet
until Neutron is finished, so the virtual machines may not actually behave
as expected for an end user deployment.
Two options exist for those without an existing Kubernetes environment:
The upstream Kubernetes community provides instructions for running Kubernetes
using Vagrant, available from:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md
The Kolla developers develop Kolla in OpenStack, using Heat to provision the
necessary servers and other resources. If you are familiar with Heat and
have a correctly configured environment available, this allows deployment
of a working Kubernetes cluster automatically. The Heat templates are
available from https://github.com/larsks/heat-kubernetes/. The templates
require at least Heat 2014.1.3 (earlier versions have a bug that will prevent
the templates from working).
Here are some simple steps to get things rolling using the Heat templates:
1. Clone the repository:
```
git clone https://github.com/larsks/heat-kubernetes/
cd heat-kubernetes
```
2. Create an appropriate image by running the get_image.sh script in this
repository. This will generate an image called `fedora-20-k8s.qcow2`.
Upload this image to Glance. You can also obtain an appropriate image from
https://fedorapeople.org/groups/heat/kolla/fedora-20-k8s.qcow2
3. Create a file `local.yaml` with settings appropriate to your OpenStack
environment. It should look something like:
```
parameters:
server_image: fedora-20-k8s
ssh_key_name: sdake
dns_nameserver: 8.8.8.8
external_network_id: 6e7e7701-46a0-49c0-9f06-ac5abc79d6ae
number_of_minions: 1
server_flavor: m1.large
```
You *must* provide settings for external_network_id and ssh_key_name; these
are local to your environment. You will probably also need to provide
a value for server_image, which should be the name (or UUID) of a Fedora 20
cloud image or derivative.
4. `heat stack-create -f kubecluster.yaml -e local.yaml my-kube-cluster`
5. Determine the ip addresses of your cluster hosts by running:
```
heat output-show my-kube-cluster kube_minions_external
```
6. Connect to the minion node with `ssh fedora@${minion-ip}`
7. On the minion node:
```
minion$ git clone http://github.com/stackforge/kolla
minion$ cd kolla
minion$ ./tools/start
```
Debugging
==========
A few commands for debugging the system.
```
$ sudo docker images
```
Lists all images that have been pulled from the upstream kollaglue repository
thus far. This can be run on the minion during the `./start` operation to
check on the download progress.
```
$ sudo docker ps -a
```
This will show all processes that docker has started. Removing the `-a` will
show only active processes. This can be run on the minion during the `./start`
operation to check that the containers are orchestrated.
```
$ sudo docker logs <containerid>
```
This shows the logging output of each service in a container. The containerid
can be obtained via the `docker ps` operation. This can be run on the minion
during the `./start` operation to debug the container.
```
$ kubectl get pods
```
This lists all pods of which Kubernetes is aware. This can be run on the
master or minion.
```
$ sudo systemctl restart kube-apiserver
$ sudo systemctl restart kube-scheduler
```
This command is needed on the master after heat finishes the creation of the
Kubernetes system (ie: my-kube-cluster is in CREATE_COMPLETE state). This is
just a workaround for a bug in kubernetes that should be fixed soon.
```
$ journalctl -f -l -xn -u kube-apiserver -u etcd -u kube-scheduler
```
This shows log output on the server for the various daemons and can be filed
in bug reports in the upstream launchpad tracker.
```
$ journalctl -f -l -xn -u kubelet.service -u kube-proxy -u docker
```
This shows log output on the minion for the various daemons and can be filed
in bug reports in the upstream launchpad tracker.
```
$ telnet minion_ip 3306
```
This shows that the Mariadb service is running on the minions. Output should
appear as follows
```
$ telnet 10.0.0.4 3306
Trying 10.0.0.4...
Connected to 10.0.0.4.
Escape character is '^]'.
5.5.39-MariaDB-wsrep
```
If the connection closes before mysql responds then the proxy is not properly
connecting to the database. This can be seen by using jounalctl and watching
for a connection error on the node that you can't connect to mysql through.
```
$ journalctl -f -l -xn -u kube-proxy
```
If you can conect though one and not the other there's probably a problem with
the overlay network. Double check that you're tunning kernel 3.16+ because
vxlan support is required. If you kernel version is good try restarting
openvswitch on both nodes. This has usually fixed the connection issues.
Directories
===========
@ -203,3 +58,28 @@ Directories
images
* k8s - contains service and pod configuration information for Kubernetes
* tools - contains different tools for interacting with Kolla
* devenv - A collection of tools and resources for managing a Kolla
development environment.
Getting Involved
================
Need a feature? Find a bug? Let us know! Contributions are much appreciated
and should follow the standard [Gerrit workflow][].
- We communicate using the #tripleo irc channel.
- File bugs, blueprints, track releases, etc on [Launchpad][].
- Attend weekly [meetings][].
- Contribute [code][]
[Gerrit workflow]: https://wiki.openstack.org/wiki/Gerrit_Workflow
[Launchpad]: https://launchpad.net/kolla
[meetings]: https://wiki.openstack.org/wiki/Meetings/Kolla
[code]: https://github.com/stackforge/kolla
Contributors
============
Check out who's [contributing][].
[contributing]: https://github.com/stackforge/kolla/graphs/contributors

143
devenv/README.md Normal file
View File

@ -0,0 +1,143 @@
A Kolla Cluster with Heat
=========================
These [Heat][] templates will deploy an *N*-node [Kolla][] cluster,
where *N* is the value of the `number_of_nodes` parameter you
specify when creating the stack.
Kolla has recently undergone a considerable design change. The details
of the design change is addressed in this [spec][]. As part of the
design change, containers share pid and networking namespaces with
the Docker host. Therefore, containers no longer connect to a docker0
bridge and have separate networking from the host. As a result, Kolla
networking has a configuration similar to:
![Image](https://raw.githubusercontent.com/stackforge/kolla/master/devenv/kollanet.png)
Sharing pid and networking namespaces is detailed in the
[super privileged containers][] concept.
The Kolla cluster is based on Fedora 21, and makes use of the
[pkilambi/docker][] [COPR][] repository for Docker packages. This
is because Kolla requires a newer version of Docker not currently
packaged in Fedora 21.
These templates are designed to work with the Icehouse or Juno
versions of Heat. If using Icehouse Heat, this [patch][] is
required to correct a bug with template validation when using the
"Fn::Join" function).
[heat]: https://wiki.openstack.org/wiki/Heat
[kolla]: https://launchpad.net/kolla
[pkilambi/docker]: https://copr.fedoraproject.org/coprs/pkilambi/docker
[copr]: https://copr.fedoraproject.org/
[spec]: https://review.openstack.org/#/c/153798/
[super privileged containers]: http://sdake.io/2015/01/28/an-atomic-upgrade-process-for-openstack-compute-nodes/
[patch]: https://review.openstack.org/#/c/121139/
Create the Glance Image
=======================
After cloning the project, run the get-image.sh script from the project's
devenv directory:
$ ./get-image.sh
The script will create a Fedora 21 image with the required modifications.
Copy the image to your Glance image store:
$ glance image-create --name "fedora-21-x86_64" \
--file /var/lib/libvirt/images/fedora-21-x86_64 \
--disk-format qcow2 --container-format bare \
--is-public True --progress
Create the Stack
================
Copy local.yaml.example to local.yaml and edit the contents to match
your deployment environment. Here is an example of a customized
local.yaml:
parameters:
ssh_key_name: admin-key
external_network_id: 028d70dd-67b8-4901-8bdd-0c62b06cce2d
dns_nameserver: 192.168.200.1
Review the parameters section of kollacluster.yaml for a full list of
configuration options. **Note:** You must provide values for:
- `ssh_key_name`
- `external_network_id`
And then create the stack, referencing that environment file:
$ heat stack-create -f kollacluster.yaml -e local.yaml kolla-cluster
Access the Kolla Nodes
======================
You can get the ip address of the Kolla nodes using the `heat
output-show` command:
$ heat output-show kolla-cluster kolla_node_external_ip
"192.168.200.86"
You can ssh into that server as the `fedora` user:
$ ssh fedora@192.168.200.86
And once logged in you can run Docker commands, etc:
$ sudo docker images
Debugging
==========
A few commands for debugging the system.
```
$ sudo docker images
```
Lists all images that have been pulled from the upstream kollaglue repository
thus far. This can be run on the node during the `./start` operation to
check on the download progress.
```
$ sudo docker ps -a
```
This will show all processes that docker has started. Removing the `-a` will
show only active processes. This can be run on the node during the `./start`
operation to check that the containers are orchestrated.
```
$ sudo docker logs <containerid>
```
This shows the logging output of each service in a container. The containerid
can be obtained via the `docker ps` operation. This can be run on the node
during the `./start` operation to debug the container.
```
$ sudo systemctl restart docker
```
Restarts the Docker service on the node.
```
$ journalctl -f -l -xn -u docker
```
This shows log output on the server for the docker daemon and can be filed
in bug reports in the upstream launchpad tracker.
```
$ telnet <NODE_IP> 3306
```
You can use telnet to test connectivity to a container. This example demonstrates
the Mariadb service is running on the node. Output should appear as follows
```
$ telnet 10.0.0.4 3306
Trying 10.0.0.4...
Connected to 10.0.0.4.
Escape character is '^]'.
5.5.39-MariaDB-wsrep
```

109
devenv/kollacluster.yaml Normal file
View File

@ -0,0 +1,109 @@
heat_template_version: 2013-05-23
description: >
This template will boot a Kolla cluster with one or more
nodes (as specified by the number_of_nodes parameter,
which defaults to "1").
parameters:
#
# REQUIRED PARAMETERS
#
ssh_key_name:
type: string
description: name of ssh key to be provisioned on the Nova instances
external_network_id:
type: string
description: uuid of a network to use for floating ip addresses
#
# OPTIONAL PARAMETERS
#
server_image:
type: string
default: fedora-21-x86_64
description: glance image used to boot the Nova instance
server_flavor:
type: string
default: m1.small
description: flavor to use when booting the Nova instance
dns_nameserver:
type: string
description: address of a dns nameserver reachable in your environment
default: 8.8.8.8
number_of_nodes:
type: string
description: how many kolla nodes to spawn
default: 1
fixed_network_cidr:
type: string
description: network range for fixed ip network
default: 10.0.0.0/24
resources:
######################################################################
#
# network resources. allocate a network and router for our server.
# it would also be possible to take advantage of existing network
# resources (and have the deployer provide network and subnet ids,
# etc, as parameters), but I wanted to minmize the amount of
# configuration necessary to make this go.
fixed_network:
type: "OS::Neutron::Net"
# This is the subnet on which we will deploy our server eth0.
fixed_subnet:
type: "OS::Neutron::Subnet"
properties:
cidr: {get_param: fixed_network_cidr}
network_id:
get_resource: fixed_network
dns_nameservers:
- get_param: dns_nameserver
# create a router attached to the external network provided as a
# parameter to this stack.
extrouter:
type: "OS::Neutron::Router"
properties:
external_gateway_info:
network:
get_param: external_network_id
# attached fixed_subnet to our extrouter router.
extrouter_inside:
type: "OS::Neutron::RouterInterface"
properties:
router_id:
get_resource: extrouter
subnet_id:
get_resource:
fixed_subnet
kolla_nodes:
type: "OS::Heat::ResourceGroup"
depends_on:
- extrouter_inside
properties:
count: {get_param: number_of_nodes}
resource_def:
type: kollanode.yaml
properties:
ssh_key_name: {get_param: ssh_key_name}
server_image: {get_param: server_image}
server_flavor: {get_param: server_flavor}
fixed_network_id: {get_resource: fixed_network}
fixed_subnet_id: {get_resource: fixed_subnet}
external_network_id: {get_param: external_network_id}
outputs:
kolla_node_internal_ip:
value: {get_attr: [kolla_nodes, kolla_node_ip_eth0]}
kolla_node_external_ip:
value: {get_attr: [kolla_nodes, kolla_node_external_ip]}

BIN
devenv/kollanet.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

199
devenv/kollanode.yaml Normal file
View File

@ -0,0 +1,199 @@
heat_template_version: 2013-05-23
description: >
This is a nested stack that defines a single Kolla node,
based on a Fedora 21 cloud image. This stack is included by
a ResourceGroup resource in the parent template (kollacluster.yaml).
parameters:
server_image:
type: string
default: fedora-21-x86_64
description: glance image used to boot the server
server_flavor:
type: string
default: m1.small
description: flavor to use when booting the server
ssh_key_name:
type: string
description: name of ssh key to be provisioned on our server
external_network_id:
type: string
description: uuid of a network to use for kolla host floating ip addresses
# The following are all generated in the parent template.
fixed_network_id:
type: string
description: Network from which to allocate fixed addresses.
fixed_subnet_id:
type: string
description: Subnet from which to allocate fixed addresses.
resources:
node_wait_handle:
type: "AWS::CloudFormation::WaitConditionHandle"
node_wait_condition:
type: "AWS::CloudFormation::WaitCondition"
depends_on:
- kolla_node
properties:
Handle:
get_resource: node_wait_handle
Timeout: "6000"
######################################################################
#
# security groups. we need to permit network traffic of various
# sorts.
#
secgroup_base:
type: "OS::Neutron::SecurityGroup"
properties:
rules:
- protocol: icmp
- protocol: tcp
port_range_min: 22
port_range_max: 22
# Use by eth1 to permit all traffic to instances.
# Let the Neutron container apply security to this traffic.
secgroup_all_open:
type: "OS::Neutron::SecurityGroup"
properties:
rules:
- protocol: icmp
- protocol: tcp
- protocol: udp
secgroup_kolla:
type: "OS::Neutron::SecurityGroup"
properties:
rules:
- protocol: tcp
port_range_min: 8773
port_range_max: 8776
- protocol: tcp
port_range_min: 6080
port_range_max: 6080
- protocol: tcp
port_range_min: 6081
port_range_max: 6081
- protocol: tcp
port_range_min: 35357
port_range_max: 35357
- protocol: tcp
port_range_min: 5000
port_range_max: 5000
- protocol: tcp
port_range_min: 9191
port_range_max: 9191
- protocol: tcp
port_range_min: 9292
port_range_max: 9292
- protocol: tcp
port_range_min: 9696
port_range_max: 9696
- protocol: tcp
port_range_min: 80
port_range_max: 80
- protocol: tcp
port_range_min: 443
port_range_max: 443
- protocol: tcp
port_range_min: 8000
port_range_max: 8000
- protocol: tcp
port_range_min: 8004
port_range_max: 8004
- protocol: tcp
port_range_min: 8003
port_range_max: 8003
- protocol: tcp
port_range_min: 8080
port_range_max: 8080
- protocol: tcp
port_range_min: 8777
port_range_max: 8777
kolla_node:
type: "OS::Nova::Server"
properties:
image:
get_param: server_image
flavor:
get_param: server_flavor
key_name:
get_param: ssh_key_name
user_data_format: RAW
user_data:
str_replace:
template: |
#!/bin/sh
yum -y upgrade
yum -y remove NetworkManager
chkconfig network on
# enable dnf command
yum -y install dnf dnf-plugins-core
# Docker packages
dnf -y copr enable pkilambi/docker
dnf -y install docker
# Start Docker
systemctl enable docker
systemctl start docker
cfn-signal -e0 --data 'OK' -r 'Setup complete' '$WAIT_HANDLE'
params:
"$WAIT_HANDLE":
get_resource: node_wait_handle
networks:
- port:
get_resource: kolla_node_eth0
- port:
get_resource: kolla_node_eth1
kolla_node_eth0:
type: "OS::Neutron::Port"
properties:
network_id:
get_param: fixed_network_id
security_groups:
- get_resource: secgroup_base
- get_resource: secgroup_kolla
fixed_ips:
- subnet_id:
get_param: fixed_subnet_id
kolla_node_eth1:
type: "OS::Neutron::Port"
properties:
network_id:
get_param: external_network_id
security_groups:
- get_resource: secgroup_all_open
kolla_node_floating:
type: "OS::Neutron::FloatingIP"
properties:
floating_network_id:
get_param: external_network_id
port_id:
get_resource: kolla_node_eth0
outputs:
kolla_node_ip_eth0:
value: {get_attr: [kolla_node_eth0, fixed_ips, 0, ip_address]}
kolla_node_external_ip:
value: {get_attr: [kolla_node_floating, floating_ip_address]}

3
devenv/local.yaml.example Normal file
View File

@ -0,0 +1,3 @@
parameters:
ssh_key_name: <YOUR_NOVA_KEYPAIR>
external_network_id: <NEUTRON_EXTERNAL_NET_ID>