Cleanup Mesos/Marathon related work

We got rid of the kolla-mesos repo:

So we don't need things in the main kolla tree
related to Mesos/Marathon

Closes-Bug: #1577951
Change-Id: Iafae63914c8a6601dc1b5b04231af8a1d2cc8642
Davanum Srinivas 7 years ago committed by Davanum Srinivas (dims)
parent 31d1271f1e
commit fa88ce2e4c
  1. 24
  2. 57
  3. 9
  4. 2
  5. 7
  6. 16
  7. 10
  8. 37
  9. 2
  10. 6
  11. 17
  12. 2
  13. 31
  14. 6
  15. 1
  16. 6
  17. 188

@ -1,24 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
RUN yum -y install \
chronos \
&& yum clean all
{% elif base_distro in ['ubuntu', 'debian'] %}
RUN apt-get install -y --no-install-recommends \
chronos \
&& apt-get clean
{% endif %}
RUN useradd --user-group chronos
CMD chronos run_jar --http_port $CHRONOS_HTTP_PORT --master $CHRONOS_MASTER --zk_hosts $CHRONOS_ZK_HOSTS
{{ include_footer }}
USER chronos

@ -1,57 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
RUN yum -y install \
marathon \
&& yum clean all
{% elif base_distro in ['ubuntu', 'debian'] %}
COPY openjdk.list /etc/apt/sources.list.d/openjdk.list
RUN apt-key adv --keyserver --recv DA1A4A13543B466853BAF164EB9B1D8886F44E2A \
&& apt-get update
# NOTE(nihilifer): Marathon package for Ubuntu depends on Oracle Java and
# there is no official way to use OpenJDK. There is no way to accept the
# license in kolla. That's why the fake package is created here and OpenJDK
# is used anyway.
# We only have to do this with Ubuntu - CentOS packages don't depend strictly
# on Oracle JDK and installing them by yum just installs OpenJDK as a
# dependency.
RUN apt-get install -y --no-install-recommends \
equivs \
gcc \
&& apt-get clean
COPY java8-runtime-headless /
RUN equivs-build java8-runtime-headless \
&& dpkg -i java8-runtime-headless_42_all.deb \
&& rm java8-runtime-headless java8-runtime-headless_42_all.deb
# NOTE(ajafo) Installation of Marathon/OpenJDK breaks cacerts file,
# so the ca-certificates-java package is purged
# and installed again as a workaround for this problem.
RUN apt-get install -y --no-install-recommends \
marathon \
openjdk-8-jre-headless \
&& dpkg --purge --force-depends ca-certificates-java \
&& apt-get install -y --no-install-recommends \
ca-certificates-java \
&& apt-get clean
{% endif %}
RUN useradd --user-group marathon \
&& chmod 755 /usr/bin/marathon
CMD ["marathon", "--no-logger"]
{{ include_footer }}
USER marathon

@ -1,9 +0,0 @@
Section: misc
Priority: optional
Standards-Version: 3.9.2
Package: java8-runtime-headless
Version: 1:42
Maintainer: Kolla Project (
Architecture: all
Description: fake Oracle Java package to block a non-free dependency

@ -1,2 +0,0 @@
# OpenJDK 8 repo
deb trusty main

@ -1,7 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}base:{{ tag }}
MAINTAINER {{ maintainer }}
RUN curl -o /usr/local/bin/mesos-dns -L \
&& chmod +x /usr/local/bin/mesos-dns
{{ include_footer }}

@ -1,16 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesosphere-base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
RUN yum -y install \
mesos \
&& yum clean all
{% elif base_distro in ['ubuntu', 'debian'] %}
RUN apt-get install -y --no-install-recommends \
mesos \
&& apt-get clean
{% endif %}

@ -1,10 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
MAINTAINER {{ maintainer }}
RUN useradd --user-group mesos
CMD ["mesos-master"]
{{ include_footer }}
USER mesos

@ -1,37 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
COPY docker.repo /etc/yum.repos.d/docker.repo
RUN rpm --import
RUN yum -y install \
docker-engine \
&& yum clean all
{% elif base_distro in ['ubuntu', 'debian'] %}
COPY docker.list /etc/apt/sources.list.d/docker.list
RUN apt-key adv --keyserver hkp:// --recv 58118E89F3A912897C070ADBF76221572C52609D \
&& apt-get update
RUN apt-get install -y --no-install-recommends \
docker-engine \
&& apt-get clean
{% endif %}
# Increase executor's timeout for pulling a Docker image to the slave
CMD ["mesos-slave"]
{{ include_footer }}
# This container should run as a root, because it's talking to the Docker
# socket directly.

@ -1,2 +0,0 @@
# Docker repo
deb ubuntu-trusty main

@ -1,6 +0,0 @@
name=Docker Repository

@ -1,17 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
RUN rpm --import
RUN rpm -Uvh
{% elif base_distro in ['ubuntu', 'debian'] %}
COPY mesosphere.list /etc/apt/sources.list.d/mesosphere.list
RUN apt-key adv --keyserver --recv 81026D0004C44CF7EF55ADF8DF7D54CBE56151BF \
&& apt-get update
{% endif %}

@ -1,2 +0,0 @@
# Mesosphere repo
deb trusty main

@ -1,31 +0,0 @@
FROM {{ namespace }}/{{ image_prefix }}mesosphere-base:{{ tag }}
MAINTAINER {{ maintainer }}
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
RUN yum -y install \
mesosphere-zookeeper \
&& yum clean all
RUN useradd --user-group zookeeper
{% elif base_distro in ['ubuntu', 'debian'] %}
RUN apt-get install -y --no-install-recommends \
zookeeper \
&& apt-get clean
{% endif %}
COPY zookeeper_sudoers /etc/sudoers.d/zookeeper_sudoers
COPY /usr/local/bin/kolla_extend_start
RUN chmod 755 /usr/local/bin/kolla_extend_start \
&& chmod 750 /etc/sudoers.d \
&& chmod 440 /etc/sudoers.d/zookeeper_sudoers \
&& usermod -a -G kolla zookeeper
ENV ZOOCFGDIR=/etc/zookeeper/conf
{{ include_footer }}
USER zookeeper

@ -1,6 +0,0 @@
# Only update permissions if permissions need to be updated
if [[ $(stat -c %U:%G /var/lib/zookeeper) != "zookeeper:zookeeper" ]]; then
sudo chown zookeeper: /var/lib/zookeeper

@ -1 +0,0 @@
%kolla ALL=(root) NOPASSWD: /bin/chown zookeeper\: /var/lib/zookeeper, /usr/bin/chown zookeeper\: /var/lib/zookeeper

@ -49,11 +49,7 @@ _PROFILE_OPTS = [
default=['cron', 'glance', 'haproxy', 'keepalived', 'keystone',
'kolla-toolbox', 'mariadb', 'memcached', 'neutron',
'nova', 'openvswitch', 'rabbitmq', 'heka'],
help='Gate images'),
default=['chronos', 'marathon', 'mesos-dns', 'mesos-master',
'mesos-slave', 'zookeeper'],
help='Mesos images')
help='Gate images')

@ -1,188 +0,0 @@
Deploy Kolla images with Mesos
Kolla deploys the containers using Ansible, however this is just one
way to deploy the containers. For example TripleO deploys Kolla
containers using Heat in-guest agents.
This specification defines the support for deploying Kolla containers
using Mesos and Marathon.
What is Mesos?
From ( Mesos "provides efficient resource
isolation and sharing across distributed applications, or frameworks".
The software enables resource sharing in a fine-grained manner,
improving cluster utilization.
What is Marathon?
From (
"A cluster-wide init and control system for services in cgroups or
Docker containers".
Adding Mesos/Marathon support to Kolla will enable those interested in
deploying OpenStack with Mesos to contribute to the Kolla community
in a more direct way.
Problem description
The current deployment (Ansible) is done somewhat serially, meaning
that some services depend on others, and the deployment is controlled
by the command line (a user). In addition to deployment, Mesos/Marathon
provides the following features that will eventually be used:
- life-cycle management: like service monitoring, restart, scaling
and rolling\restarts\upgrades
- constraints [1]: the Marathon scheduler will be used to more
effectively place containers (esp. during scaling/recovery)
- integration with core infrastructure services like DNS, Load
Balancing, Service Discovery and Service components.
In order to reuse a large amount of functionality, it would be best
to use an existing framework that provides a proven stable and
mature solution.
Given that Mesos/Marathon is used and tested at scale by many large
companies, it will give operators the confidence to adopt
OpenStack to meet any scaling requirements they need.
Marathon [2] will be used to manage the containers. Marathon is a
framework that runs on top of Mesos and it is for long running
Part of this change is to start all the containers at the same time
(in parallel) so that there are as few dependencies from the
deployment tool’s point of view. This should enable a couple of things:
- faster initial deployment
- reduce unnecessary restarts during upgrades
- make each container more self sufficient
Proposed change
- Add a deployment specific git repo (kolla-mesos) to contain the
Mesos/Marathon specific deployment code and boot strapping.
- Enhance Kolla container API (config.json) to permit loading
of custom startup script while maintaining immutability with copy_once.
- Implement an all in one (AIO) basic OpenStack
- Implement a separate controller/compute setup similar to the Ansible one.
- Throughout add docs to assist users and contributors/reviewers.
At first, Mesos/Marathon/Zookeeper bootstrapping will be done by
setting up docker container. Later, bootstrapping will be handled by Ironic/PXE
(the aim is to be practical and do what is easiest for the AIO).
Dependancy management
Instead of the serialising the dependant steps, each container is
started and only actually starts the service if the requirements are
These dependencies will come in the form of:
- service discovery (service X needs service Y running)
Note: that Marathon DNS and LB can be self-configured based on service
registry information.
To achieve this the container also needs to register itself once
it has started.
- checking to see if service configuration is complete
(has keystone got the service user that is required, is the DB
schema complete, etc..)
Use Zookeeper to watch for these configuration steps.
One time tasks
Ansible runs a number of scripts to setup the database, keystone etc.
These can be run as a Mesos Executor (command line run in the
container of choice).
Security impact
Mesos and Marathon are mature products used by various companies in
production. The central configuration storage will require careful
security risk assessment. The deployed OpenStack’s security should not
be affected by the deployment tool.
Performance Impact
Given that the Mesos slaves are distributed and all containers will be
started in parallel, the deployment *may* be faster, though this is
not the main focus.
Kubernetes was evaluated by the Kolla team 6 months ago and found to
not work at that time as it did not support net=host and pid=host
features of docker. Since then it has developed these features, if
Mesos/Marathon fails to produce results, then going back to kubernetes
is an option. However at the time of writing this Mesos/Marathon was
deemed to be more mature and stable.
Primary Assignee(s)
Angus Salkeld (asalkeld)
Kirill Proskurin (kproskurin)
Michal Rostecki (nihilifer)
Other contributor(s):
Harm Weites (harmw)
Jeff Peeler (jpeeler)
Michal Jastrzebski (inc0)
Sam Yaple (SamYaple)
Steven Dake (sdake)
<Please add your name here if you are getting involved in kolla-mesos>
Target Milestone for completion:
Work Items
1. Allow a custom startup script to run (change in Kolla)
2. Add startup scripts to kolla-mesos to read config from zookeeper
instead of bindmounted directory. Propose oslo.config changes to
use this method (oslo work done in parallel, initially this will be
done in the startup script).
3. Add startup scripts for service discovery so that services only
start once their needs are fulfilled.
a. register a service once a service is running
b. wait for dependent services if they are needed before starting
a service.
c. DNS and LB self-configuration based on service registry information
5. Add bootstrapping code to install Marathon, Zookeeper,
Mesos master and slave.
6. Add calls to to marathon to deploy containers.
7. Add support for kolla-mesos to kolla-cli.
Functional tests will be implemented in the OpenStack check/gating system to
automatically check that the Mesos/Marathon deployment works for an AIO environment.
Documentation Impact
A quick start guide will be written to explain how to deploy.
A develop guide will be written on how to contribute and how the deployment works.
- [1]
- [2]