Container Infrastructure Management Service for OpenStack
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

24 KiB

Developer Quick-Start

This is a quick walkthrough to get you started developing code for magnum. This assumes you are already familiar with submitting code reviews to an OpenStack project.

Setup Dev Environment

Install OS-specific prerequisites:

# Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y python-dev libssl-dev libxml2-dev \
                        libmysqlclient-dev libxslt-dev libpq-dev git \
                        libffi-dev gettext build-essential python3.4-dev

# Fedora/RHEL:
sudo yum install -y python-devel openssl-devel mysql-devel \
                    libxml2-devel libxslt-devel postgresql-devel git \
                    libffi-devel gettext gcc

# openSUSE/SLE 12:
sudo zypper --non-interactive install git libffi-devel \
                    libmysqlclient-devel libopenssl-devel libxml2-devel \
                    libxslt-devel postgresql-devel python-devel \

Install pip:

curl -s | sudo python

Install common prerequisites:

sudo pip install virtualenv flake8 tox testrepository git-review

You may need to explicitly upgrade virtualenv if you've installed the one from your OS distribution and it is too old (tox will complain). You can upgrade it individually, if you need to:

sudo pip install -U virtualenv

Magnum source code should be pulled directly from git:

# from your home or source directory
cd ~
git clone
cd magnum

All unit tests should be run using tox. To run magnum's entire test suite:

# run all tests (unit and pep8)

To run a specific test, use a positional argument for the unit tests:

# run a specific test for Python 2.7
tox -epy27 -- test_conductor

You may pass options to the test programs using positional arguments:

# run all the Python 2.7 unit tests (in parallel!)
tox -epy27 -- --parallel

To run only the pep8/flake8 syntax and style checks:

tox -epep8

To run unit test coverage and check percentage of code covered:

tox -e cover

To discover and interact with templates, please refer to

Exercising the Services Using Devstack

Devstack can be configured to enable magnum support. It is easy to develop magnum with the devstack environment. Magnum depends on nova, glance, heat and neutron to create and schedule virtual machines to simulate bare-metal (full bare-metal support is under active development).

NOTE: Running devstack within a virtual machine with magnum enabled is not recommended at this time.

This session has only been tested on Ubuntu 14.04 (Trusty) and Fedora 20/21. We recommend users to select one of them if it is possible.

For in-depth guidance on adding magnum manually to a devstack instance, please refer to the

Clone devstack:

# Create a root directory for devstack if needed
sudo mkdir -p /opt/stack
sudo chown $USER /opt/stack

git clone /opt/stack/devstack

We will run devstack with minimal local.conf settings required to enable magnum, heat, and neutron (neutron is enabled by default in devstack since Kilo, and heat is enabled by the magnum plugin):

cat > /opt/stack/devstack/local.conf << END
# magnum requires the following to be set correctly
enable_plugin magnum
# Enable barbican service and use it to store TLS certificates
# For details
enable_plugin barbican
enable_plugin neutron-lbaas

NOTE: Update PUBLIC_INTERFACE as appropriate for your system.

Optionally, you can enable ceilometer in devstack. If ceilometer is enabled, magnum will periodically send metrics to ceilometer:

cat >> /opt/stack/devstack/local.conf << END
enable_plugin ceilometer

If you want to deploy Docker Registry 2.0 in your bay, you should enable swift in devstack:

cat >> /opt/stack/devstack/local.conf << END
enable_service s-proxy
enable_service s-object
enable_service s-container
enable_service s-account

More devstack configuration information can be found at

More neutron configuration information can be found at

Run devstack:

cd /opt/stack/devstack

NOTE: This will take a little extra time when the Fedora Atomic micro-OS image is downloaded for the first time.

At this point, two magnum process (magnum-api and magnum-conductor) will be running on devstack screens. If you make some code changes and want to test their effects, just stop and restart magnum-api and/or magnum-conductor.

Prepare your session to be able to use the various openstack clients including magnum, neutron, and glance. Create a new shell, and source the devstack openrc script:

source /opt/stack/devstack/openrc admin admin

Magnum has been tested with the Fedora Atomic micro-OS and CoreOS. Magnum will likely work with other micro-OS platforms, but each requires individual support in the heat template.

The Fedora Atomic micro-OS image will automatically be added to glance. You can add additional images manually through glance. To verify the image created when installing devstack use:

glance -v image-list

| ID                                   | Name                            | Disk_format | Container_format | Size      | Status | Owner                            |
| 090de3a2-2c0c-42d5-b5a3-cfcddd6d011b | cirros-0.3.4-x86_64-uec         | ami         | ami              | 25165824  | active | f98b9727094d40c78b1ed40e3bc91e80 |
| 9501d296-f0aa-4c0e-bc24-2a680f8741f0 | cirros-0.3.4-x86_64-uec-kernel  | aki         | aki              | 4979632   | active | f98b9727094d40c78b1ed40e3bc91e80 |
| 01478d1a-59e0-4f36-b69e-0eaf5821ee46 | cirros-0.3.4-x86_64-uec-ramdisk | ari         | ari              | 3740163   | active | f98b9727094d40c78b1ed40e3bc91e80 |
| f14d6ee3-9e53-4f22-ba42-44e95810c294 | fedora-atomic-latest            | qcow2       | bare             | 507928064 | active | f98b9727094d40c78b1ed40e3bc91e80 |

To list the available commands and resources for magnum, use:

magnum help

To list out the health of the internal services, namely conductor, of magnum, use:

magnum service-list

| id | host                               | binary           | state |
| 1  | | magnum-conductor | up    |

Create a keypair for use with the baymodel:

test -f ~/.ssh/ || ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
nova keypair-add --pub-key ~/.ssh/ testkey

Building a Kubernetes Bay - Based on Fedora Atomic

Create a baymodel. This is similar in nature to a flavor and describes to magnum how to construct the bay. The baymodel specifies a Fedora Atomic image so the bays which use this baymodel will be based on Fedora Atomic. The coe (Container Orchestration Engine) and keypair need to be specified as well:

magnum baymodel-create --name k8sbaymodel \
                       --image-id fedora-atomic-latest \
                       --keypair-id testkey \
                       --external-network-id public \
                       --dns-nameserver \
                       --flavor-id m1.small \
                       --docker-volume-size 5 \
                       --network-driver flannel \
                       --coe kubernetes

Create a bay. Use the baymodel name as a template for bay creation. This bay will result in one master kubernetes node and one minion node:

magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1

Bays will have an initial status of CREATE_IN_PROGRESS. Magnum will update the status to CREATE_COMPLETE when it is done creating the bay. Do not create containers, pods, services, or replication controllers before magnum finishes creating the bay. They will likely not be created, and may cause magnum to become confused.

The existing bays can be listed as follows:

magnum bay-list

| uuid                                 | name    | node_count | status          |
| 9dccb1e6-02dc-4e2b-b897-10656c5339ce | k8sbay  | 1          | CREATE_COMPLETE |

More detailed information for a given bay is obtained via:

magnum bay-show k8sbay

After a bay is created, you can dynamically add/remove node(s) to/from the bay by updating the node_count attribute. For example, to add one more node:

magnum bay-update k8sbay replace node_count=2

Bays in the process of updating will have a status of UPDATE_IN_PROGRESS. Magnum will update the status to UPDATE_COMPLETE when it is done updating the bay.

NOTE: Reducing node_count will remove all the existing pods on the nodes that are deleted. If you choose to reduce the node_count, magnum will first try to remove empty nodes with no pods running on them. If you reduce node_count by more than the number of empty nodes, magnum must remove nodes that have running pods on them. This action will delete those pods. We strongly recommend using a replication controller before reducing the node_count so any removed pods can be automatically recovered on your remaining nodes.

Heat can be used to see detailed information on the status of a stack or specific bay:

To check the list of all bay stacks:

heat stack-list

To check an individual bay's stack:

heat stack-show <stack-name or stack_id>

Monitoring bay status in detail (e.g., creating, updating):

BAY_HEAT_NAME=$(heat stack-list | awk "/\sk8sbay-/{print \$4}")
heat resource-list ${BAY_HEAT_NAME}

Building a Kubernetes Bay - Based on CoreOS

You can create a Kubernetes bay based on CoreOS as an alternative to Atomic. First, download the official CoreOS image:

bunzip2 coreos_production_openstack_image.img.bz2

Upload the image to glance:

glance image-create --name CoreOS  \
                    --visibility public \
                    --disk-format=qcow2 \
                    --container-format=bare \
                    --os-distro=coreos \

Create a CoreOS Kubernetes baymodel, which is similar to the Atomic Kubernetes baymodel, except for pointing to a different image:

magnum baymodel-create --name k8sbaymodel-coreos \
                       --image-id CoreOS \
                       --keypair-id testkey \
                       --external-network-id public \
                       --dns-nameserver \
                       --flavor-id m1.small \
                       --network-driver flannel \
                       --coe kubernetes \

Create a CoreOS Kubernetes bay. Use the CoreOS baymodel as a template for bay creation:

magnum bay-create --name k8sbay \
                  --baymodel k8sbaymodel-coreos \
                  --node-count 2

Using Kubernetes Bay

NOTE: For the following examples, only one minion node is required in the k8s bay created previously.

Kubernetes provides a number of examples you can use to check that things are working. You may need to clone kubernetes using:

tar -xvzf kubernetes.tar.gz

NOTE: We do not need to install Kubernetes, we just need the example file from the tarball.

Here's how to set up the replicated redis example. First, create a pod for the redis-master:

cd kubernetes/examples/redis
magnum pod-create --manifest ./redis-master.yaml --bay k8sbay

Now create a service to provide a discoverable endpoint for the redis sentinels in the cluster:

magnum coe-service-create --manifest ./redis-sentinel-service.yaml --bay k8sbay

To make it a replicated redis cluster create replication controllers for the redis slaves and sentinels:

sed -i 's/\(replicas: \)1/\1 2/' redis-controller.yaml
magnum rc-create --manifest ./redis-controller.yaml --bay k8sbay

sed -i 's/\(replicas: \)1/\1 2/' redis-sentinel-controller.yaml
magnum rc-create --manifest ./redis-sentinel-controller.yaml --bay k8sbay

Full lifecycle and introspection operations for each object are supported. For example, magnum bay-create, magnum baymodel-delete, magnum rc-show, magnum coe-service-list.

Now there are four redis instances (one master and three slaves) running across the bay, replicating data between one another.

Run the bay-show command to get the IP of the bay host on which the redis-master is running:

magnum bay-show k8sbay

| Property           | Value                                                      |
| status             | CREATE_COMPLETE                                            |
| uuid               | 481685d2-bc16-4daf-9aac-9e830c7da3f7                       |
| status_reason      | Stack CREATE completed successfully                        |
| created_at         | 2015-09-22T20:02:39+00:00                                  |
| updated_at         | 2015-09-22T20:05:00+00:00                                  |
| bay_create_timeout | 0                                                          |
| api_address        |                                         |
| baymodel_id        | 194a4b7e-0125-4956-8660-7551469ae1ed                       |
| node_count         | 1                                                          |
| node_addresses     | [u'']                                         |
| master_count       | 1                                                          |
| discovery_url      | |
| name               | k8sbay                                                     |

The output here indicates the redis-master is running on the bay host with IP address To access the redis master:

ssh minion@
REDIS_ID=$(sudo docker ps | grep redis:v1 | grep k8s_master | awk '{print $1}')
sudo docker exec -i -t $REDIS_ID redis-cli> set replication:test true

exit  # Log out of the host

Log into one of the other container hosts and access a redis slave from it. You can use nova list to enumerate the kube-minions. For this example we will use the same host as above:

ssh minion@
REDIS_ID=$(sudo docker ps | grep redis:v1 | grep k8s_redis | awk '{print $1}')
sudo docker exec -i -t $REDIS_ID redis-cli> get replication:test

exit  # Log out of the host

Additional useful commands from a given minion:

sudo docker ps  # View Docker containers on this minion
kubectl get po  # Get pods
kubectl get rc  # Get replication controllers
kubectl get svc  # Get services
kubectl get nodes  # Get nodes

After you finish using the bay, you want to delete it. A bay can be deleted as follows:

magnum bay-delete k8sbay

Building and Using a Swarm Bay

Create a baymodel. It is very similar to the Kubernetes baymodel, except for the absence of some Kubernetes-specific arguments and the use of 'swarm' as the coe:

magnum baymodel-create --name swarmbaymodel \
                       --image-id fedora-atomic-latest \
                       --keypair-id testkey \
                       --external-network-id public \
                       --dns-nameserver \
                       --flavor-id m1.small \
                       --docker-volume-size 5 \
                       --coe swarm

NOTE: If you are using Magnum behind a firewall then see:

Finally, create the bay. Use the baymodel 'swarmbaymodel' as a template for bay creation. This bay will result in one swarm manager node and two extra agent nodes:

magnum bay-create --name swarmbay --baymodel swarmbaymodel --node-count 2

Now that we have a swarm bay we can start interacting with it:

magnum bay-show swarmbay

| Property      | Value                                    |
| status        | CREATE_COMPLETE                          |
| uuid          | eda91c1e-6103-45d4-ab09-3f316310fa8e     |
| created_at    | 2015-04-20T19:05:27+00:00                |
| updated_at    | 2015-04-20T19:06:08+00:00                |
| baymodel_id   | a93ee8bd-fec9-4ea7-ac65-c66c1dba60af     |
| node_count    | 2                                        |
| discovery_url |                                          |
| name          | swarmbay                                 |

Next we will create a container in this bay. This container will ping the address four times:

magnum container-create --name test-container \
                        --image \
                        --bay swarmbay \
                        --command "ping -c 4"

| Property   | Value                                  |
| uuid       | 25485358-ae9b-49d1-a1e1-1af0a7c3f911   |
| links      | ...                                    |
| bay_uuid   | eda91c1e-6103-45d4-ab09-3f316310fa8e   |
| updated_at | None                                   |
| image      | cirros                                 |
| command    | ping -c 4                      |
| created_at | 2015-04-22T20:21:11+00:00              |
| name       | test-container                         |

At this point the container exists but it has not been started yet. To start it and check its output run the following:

magnum container-start test-container
magnum container-logs test-container

PING ( 56 data bytes
64 bytes from seq=0 ttl=40 time=25.513 ms
64 bytes from seq=1 ttl=40 time=25.348 ms
64 bytes from seq=2 ttl=40 time=25.226 ms
64 bytes from seq=3 ttl=40 time=25.275 ms

--- ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 25.226/25.340/25.513 ms

Now that we're done with the container we can delete it:

magnum container-delete test-container

Building and Using a Mesos Bay

Provisioning a mesos bay requires a Ubuntu-based image with some packages pre-installed. To build and upload such image, please refer to

Alternatively, you can download and upload a pre-built image:

glance image-create --name ubuntu-mesos --visibility public \
                    --disk-format=qcow2 --container-format=bare \
                    --os-distro=ubuntu --file=ubuntu-14.04.3-mesos-0.25.0.qcow2

Then, create a baymodel by using 'mesos' as the coe, with the rest of arguments similar to the Kubernetes baymodel:

magnum baymodel-create --name mesosbaymodel --image-id ubuntu-mesos \
                       --keypair-id testkey \
                       --external-network-id public \
                       --dns-nameserver \
                       --flavor-id m1.small \
                       --coe mesos

Finally, create the bay. Use the baymodel 'mesosbaymodel' as a template for bay creation. This bay will result in one mesos master node and two mesos slave nodes:

magnum bay-create --name mesosbay --baymodel mesosbaymodel --node-count 2

Now that we have a mesos bay we can start interacting with it. First we need to make sure the bay's status is 'CREATE_COMPLETE':

$ magnum bay-show mesosbay
| Property           | Value                                |
| status             | CREATE_COMPLETE                      |
| uuid               | ff727f0d-72ca-4e2b-9fef-5ec853d74fdf |
| status_reason      | Stack CREATE completed successfully  |
| created_at         | 2015-06-09T20:21:43+00:00            |
| updated_at         | 2015-06-09T20:28:18+00:00            |
| bay_create_timeout | 0                                    |
| api_address        |                         |
| baymodel_id        | 92dbda62-32d4-4435-88fc-8f42d514b347 |
| node_count         | 2                                    |
| node_addresses     | [u'', u'']   |
| master_count       | 1                                    |
| discovery_url      | None                                 |
| name               | mesosbay                             |

Next we will create a container in this bay by using the REST API of Marathon. This container will ping the address

$ cat > mesos.json << END
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "cirros"
  "id": "ubuntu",
  "instances": 1,
  "cpus": 0.5,
  "mem": 512,
  "uris": [],
  "cmd": "ping"
$ MASTER_IP=$(magnum bay-show mesosbay | awk '/ api_address /{print $4}')
$ curl -X POST -H "Content-Type: application/json" \
    http://${MASTER_IP}:8080/v2/apps -d@mesos.json

To check application and task status:

$ curl http://${MASTER_IP}:8080/v2/apps
$ curl http://${MASTER_IP}:8080/v2/tasks

You can access to the Mesos web page at http://<master>:5050/ and Marathon web console at http://<master>:8080/.

Building Developer Documentation

To build the documentation locally (e.g., to test documentation changes before uploading them for review) chdir to the magnum root folder and run tox:

tox -edocs

NOTE: The first time you run this will take some extra time as it creates a virtual environment to run in.

When complete, the documentation can be accessed from: