Cleanup Mesos/Marathon related work
We got rid of the kolla-mesos repo: http://git.openstack.org/cgit/openstack/kolla-mesos/tree/README.rst So we don't need things in the main kolla tree related to Mesos/Marathon Closes-Bug: #1577951 Change-Id: Iafae63914c8a6601dc1b5b04231af8a1d2cc8642
This commit is contained in:
Davanum Srinivas
Davanum Srinivas (dims)
parent
31d1271f1e
commit
fa88ce2e4c
@ -1,24 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
RUN yum -y install \
|
|
||||||
chronos \
|
|
||||||
&& yum clean all
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
chronos \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
RUN useradd --user-group chronos
|
|
||||||
|
|
||||||
CMD chronos run_jar --http_port $CHRONOS_HTTP_PORT --master $CHRONOS_MASTER --zk_hosts $CHRONOS_ZK_HOSTS
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
|
|
||||||
USER chronos
|
|
||||||
@ -1,57 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
RUN yum -y install \
|
|
||||||
marathon \
|
|
||||||
&& yum clean all
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
COPY openjdk.list /etc/apt/sources.list.d/openjdk.list
|
|
||||||
|
|
||||||
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv DA1A4A13543B466853BAF164EB9B1D8886F44E2A \
|
|
||||||
&& apt-get update
|
|
||||||
|
|
||||||
# NOTE(nihilifer): Marathon package for Ubuntu depends on Oracle Java and
|
|
||||||
# there is no official way to use OpenJDK. There is no way to accept the
|
|
||||||
# license in kolla. That's why the fake package is created here and OpenJDK
|
|
||||||
# is used anyway.
|
|
||||||
# We only have to do this with Ubuntu - CentOS packages don't depend strictly
|
|
||||||
# on Oracle JDK and installing them by yum just installs OpenJDK as a
|
|
||||||
# dependency.
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
equivs \
|
|
||||||
gcc \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
COPY java8-runtime-headless /
|
|
||||||
|
|
||||||
RUN equivs-build java8-runtime-headless \
|
|
||||||
&& dpkg -i java8-runtime-headless_42_all.deb \
|
|
||||||
&& rm java8-runtime-headless java8-runtime-headless_42_all.deb
|
|
||||||
|
|
||||||
# NOTE(ajafo) Installation of Marathon/OpenJDK breaks cacerts file,
|
|
||||||
# so the ca-certificates-java package is purged
|
|
||||||
# and installed again as a workaround for this problem.
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
marathon \
|
|
||||||
openjdk-8-jre-headless \
|
|
||||||
&& dpkg --purge --force-depends ca-certificates-java \
|
|
||||||
&& apt-get install -y --no-install-recommends \
|
|
||||||
ca-certificates-java \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
RUN useradd --user-group marathon \
|
|
||||||
&& chmod 755 /usr/bin/marathon
|
|
||||||
|
|
||||||
CMD ["marathon", "--no-logger"]
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
|
|
||||||
USER marathon
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
Section: misc
|
|
||||||
Priority: optional
|
|
||||||
Standards-Version: 3.9.2
|
|
||||||
|
|
||||||
Package: java8-runtime-headless
|
|
||||||
Version: 1:42
|
|
||||||
Maintainer: Kolla Project (https://launchpad.net/kolla)
|
|
||||||
Architecture: all
|
|
||||||
Description: fake Oracle Java package to block a non-free dependency
|
|
||||||
@ -1,2 +0,0 @@
|
|||||||
# OpenJDK 8 repo
|
|
||||||
deb http://ppa.launchpad.net/openjdk-r/ppa/ubuntu trusty main
|
|
||||||
@ -1,7 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
RUN curl -o /usr/local/bin/mesos-dns -L https://github.com/mesosphere/mesos-dns/releases/download/v0.5.1/mesos-dns-v0.5.1-linux-amd64 \
|
|
||||||
&& chmod +x /usr/local/bin/mesos-dns
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
@ -1,16 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesosphere-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
RUN yum -y install \
|
|
||||||
mesos \
|
|
||||||
&& yum clean all
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
mesos \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
@ -1,10 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
RUN useradd --user-group mesos
|
|
||||||
|
|
||||||
CMD ["mesos-master"]
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
|
|
||||||
USER mesos
|
|
||||||
@ -1,37 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesos-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
COPY docker.repo /etc/yum.repos.d/docker.repo
|
|
||||||
|
|
||||||
RUN rpm --import https://yum.dockerproject.org/gpg
|
|
||||||
|
|
||||||
RUN yum -y install \
|
|
||||||
docker-engine \
|
|
||||||
&& yum clean all
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
COPY docker.list /etc/apt/sources.list.d/docker.list
|
|
||||||
|
|
||||||
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D \
|
|
||||||
&& apt-get update
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
docker-engine \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
ENV MESOS_CONTAINERIZERS docker
|
|
||||||
# Increase executor's timeout for pulling a Docker image to the slave
|
|
||||||
# https://mesosphere.github.io/marathon/docs/native-docker.html
|
|
||||||
ENV MESOS_EXECUTOR_REGISTRATION_TIMEOUT 5mins
|
|
||||||
|
|
||||||
CMD ["mesos-slave"]
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
|
|
||||||
# This container should run as a root, because it's talking to the Docker
|
|
||||||
# socket directly.
|
|
||||||
@ -1,2 +0,0 @@
|
|||||||
# Docker repo
|
|
||||||
deb http://apt.dockerproject.org/repo ubuntu-trusty main
|
|
||||||
@ -1,6 +0,0 @@
|
|||||||
[dockerrepo]
|
|
||||||
name=Docker Repository
|
|
||||||
baseurl=https://yum.dockerproject.org/repo/main/centos/7
|
|
||||||
enabled=1
|
|
||||||
gpgcheck=1
|
|
||||||
gpgkey=https://yum.dockerproject.org/gpg
|
|
||||||
@ -1,17 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
RUN rpm --import http://repos.mesosphere.io/el/RPM-GPG-KEY-mesosphere
|
|
||||||
|
|
||||||
RUN rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
COPY mesosphere.list /etc/apt/sources.list.d/mesosphere.list
|
|
||||||
|
|
||||||
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv 81026D0004C44CF7EF55ADF8DF7D54CBE56151BF \
|
|
||||||
&& apt-get update
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
@ -1,2 +0,0 @@
|
|||||||
# Mesosphere repo
|
|
||||||
deb http://repos.mesosphere.com/ubuntu trusty main
|
|
||||||
@ -1,31 +0,0 @@
|
|||||||
FROM {{ namespace }}/{{ image_prefix }}mesosphere-base:{{ tag }}
|
|
||||||
MAINTAINER {{ maintainer }}
|
|
||||||
|
|
||||||
{% if base_distro in ['centos', 'fedora', 'oraclelinux', 'rhel'] %}
|
|
||||||
|
|
||||||
RUN yum -y install \
|
|
||||||
mesosphere-zookeeper \
|
|
||||||
&& yum clean all
|
|
||||||
|
|
||||||
RUN useradd --user-group zookeeper
|
|
||||||
|
|
||||||
{% elif base_distro in ['ubuntu', 'debian'] %}
|
|
||||||
|
|
||||||
RUN apt-get install -y --no-install-recommends \
|
|
||||||
zookeeper \
|
|
||||||
&& apt-get clean
|
|
||||||
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
COPY zookeeper_sudoers /etc/sudoers.d/zookeeper_sudoers
|
|
||||||
COPY extend_start.sh /usr/local/bin/kolla_extend_start
|
|
||||||
RUN chmod 755 /usr/local/bin/kolla_extend_start \
|
|
||||||
&& chmod 750 /etc/sudoers.d \
|
|
||||||
&& chmod 440 /etc/sudoers.d/zookeeper_sudoers \
|
|
||||||
&& usermod -a -G kolla zookeeper
|
|
||||||
|
|
||||||
ENV ZOOCFGDIR=/etc/zookeeper/conf
|
|
||||||
|
|
||||||
{{ include_footer }}
|
|
||||||
|
|
||||||
USER zookeeper
|
|
||||||
@ -1,6 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Only update permissions if permissions need to be updated
|
|
||||||
if [[ $(stat -c %U:%G /var/lib/zookeeper) != "zookeeper:zookeeper" ]]; then
|
|
||||||
sudo chown zookeeper: /var/lib/zookeeper
|
|
||||||
fi
|
|
||||||
@ -1 +0,0 @@
|
|||||||
%kolla ALL=(root) NOPASSWD: /bin/chown zookeeper\: /var/lib/zookeeper, /usr/bin/chown zookeeper\: /var/lib/zookeeper
|
|
||||||
@ -49,11 +49,7 @@ _PROFILE_OPTS = [
|
|||||||
default=['cron', 'glance', 'haproxy', 'keepalived', 'keystone',
|
default=['cron', 'glance', 'haproxy', 'keepalived', 'keystone',
|
||||||
'kolla-toolbox', 'mariadb', 'memcached', 'neutron',
|
'kolla-toolbox', 'mariadb', 'memcached', 'neutron',
|
||||||
'nova', 'openvswitch', 'rabbitmq', 'heka'],
|
'nova', 'openvswitch', 'rabbitmq', 'heka'],
|
||||||
help='Gate images'),
|
help='Gate images')
|
||||||
cfg.ListOpt('mesos',
|
|
||||||
default=['chronos', 'marathon', 'mesos-dns', 'mesos-master',
|
|
||||||
'mesos-slave', 'zookeeper'],
|
|
||||||
help='Mesos images')
|
|
||||||
]
|
]
|
||||||
|
|
||||||
_CLI_OPTS = [
|
_CLI_OPTS = [
|
||||||
|
|||||||
@ -1,188 +0,0 @@
|
|||||||
==============================
|
|
||||||
Deploy Kolla images with Mesos
|
|
||||||
==============================
|
|
||||||
|
|
||||||
https://blueprints.launchpad.net/kolla/+spec/mesos
|
|
||||||
|
|
||||||
Kolla deploys the containers using Ansible, however this is just one
|
|
||||||
way to deploy the containers. For example TripleO deploys Kolla
|
|
||||||
containers using Heat in-guest agents.
|
|
||||||
|
|
||||||
This specification defines the support for deploying Kolla containers
|
|
||||||
using Mesos and Marathon.
|
|
||||||
|
|
||||||
What is Mesos?
|
|
||||||
From (http://mesos.apache.org/) Mesos "provides efficient resource
|
|
||||||
isolation and sharing across distributed applications, or frameworks".
|
|
||||||
The software enables resource sharing in a fine-grained manner,
|
|
||||||
improving cluster utilization.
|
|
||||||
|
|
||||||
What is Marathon?
|
|
||||||
From (https://mesosphere.github.io/marathon/):
|
|
||||||
"A cluster-wide init and control system for services in cgroups or
|
|
||||||
Docker containers".
|
|
||||||
|
|
||||||
Adding Mesos/Marathon support to Kolla will enable those interested in
|
|
||||||
deploying OpenStack with Mesos to contribute to the Kolla community
|
|
||||||
in a more direct way.
|
|
||||||
|
|
||||||
Problem description
|
|
||||||
===================
|
|
||||||
|
|
||||||
The current deployment (Ansible) is done somewhat serially, meaning
|
|
||||||
that some services depend on others, and the deployment is controlled
|
|
||||||
by the command line (a user). In addition to deployment, Mesos/Marathon
|
|
||||||
provides the following features that will eventually be used:
|
|
||||||
|
|
||||||
- life-cycle management: like service monitoring, restart, scaling
|
|
||||||
and rolling\restarts\upgrades
|
|
||||||
- constraints [1]: the Marathon scheduler will be used to more
|
|
||||||
effectively place containers (esp. during scaling/recovery)
|
|
||||||
- integration with core infrastructure services like DNS, Load
|
|
||||||
Balancing, Service Discovery and Service components.
|
|
||||||
|
|
||||||
In order to reuse a large amount of functionality, it would be best
|
|
||||||
to use an existing framework that provides a proven stable and
|
|
||||||
mature solution.
|
|
||||||
Given that Mesos/Marathon is used and tested at scale by many large
|
|
||||||
companies, it will give operators the confidence to adopt
|
|
||||||
OpenStack to meet any scaling requirements they need.
|
|
||||||
|
|
||||||
Marathon [2] will be used to manage the containers. Marathon is a
|
|
||||||
framework that runs on top of Mesos and it is for long running
|
|
||||||
services.
|
|
||||||
|
|
||||||
Part of this change is to start all the containers at the same time
|
|
||||||
(in parallel) so that there are as few dependencies from the
|
|
||||||
deployment tool’s point of view. This should enable a couple of things:
|
|
||||||
- faster initial deployment
|
|
||||||
- reduce unnecessary restarts during upgrades
|
|
||||||
- make each container more self sufficient
|
|
||||||
|
|
||||||
Proposed change
|
|
||||||
===============
|
|
||||||
|
|
||||||
- Add a deployment specific git repo (kolla-mesos) to contain the
|
|
||||||
Mesos/Marathon specific deployment code and boot strapping.
|
|
||||||
- Enhance Kolla container API (config.json) to permit loading
|
|
||||||
of custom startup script while maintaining immutability with copy_once.
|
|
||||||
- Implement an all in one (AIO) basic OpenStack
|
|
||||||
- Implement a separate controller/compute setup similar to the Ansible one.
|
|
||||||
- Throughout add docs to assist users and contributors/reviewers.
|
|
||||||
|
|
||||||
Bootstrapping:
|
|
||||||
--------------
|
|
||||||
|
|
||||||
At first, Mesos/Marathon/Zookeeper bootstrapping will be done by
|
|
||||||
setting up docker container. Later, bootstrapping will be handled by Ironic/PXE
|
|
||||||
(the aim is to be practical and do what is easiest for the AIO).
|
|
||||||
|
|
||||||
Dependancy management
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
Instead of the serialising the dependant steps, each container is
|
|
||||||
started and only actually starts the service if the requirements are
|
|
||||||
fulfilled.
|
|
||||||
|
|
||||||
These dependencies will come in the form of:
|
|
||||||
|
|
||||||
- service discovery (service X needs service Y running)
|
|
||||||
Note: that Marathon DNS and LB can be self-configured based on service
|
|
||||||
registry information.
|
|
||||||
To achieve this the container also needs to register itself once
|
|
||||||
it has started.
|
|
||||||
- checking to see if service configuration is complete
|
|
||||||
(has keystone got the service user that is required, is the DB
|
|
||||||
schema complete, etc..)
|
|
||||||
Use Zookeeper to watch for these configuration steps.
|
|
||||||
|
|
||||||
One time tasks
|
|
||||||
--------------
|
|
||||||
Ansible runs a number of scripts to setup the database, keystone etc.
|
|
||||||
These can be run as a Mesos Executor (command line run in the
|
|
||||||
container of choice).
|
|
||||||
|
|
||||||
Security impact
|
|
||||||
---------------
|
|
||||||
|
|
||||||
Mesos and Marathon are mature products used by various companies in
|
|
||||||
production. The central configuration storage will require careful
|
|
||||||
security risk assessment. The deployed OpenStack’s security should not
|
|
||||||
be affected by the deployment tool.
|
|
||||||
|
|
||||||
Performance Impact
|
|
||||||
------------------
|
|
||||||
|
|
||||||
Given that the Mesos slaves are distributed and all containers will be
|
|
||||||
started in parallel, the deployment *may* be faster, though this is
|
|
||||||
not the main focus.
|
|
||||||
|
|
||||||
Alternatives
|
|
||||||
------------
|
|
||||||
|
|
||||||
Kubernetes was evaluated by the Kolla team 6 months ago and found to
|
|
||||||
not work at that time as it did not support net=host and pid=host
|
|
||||||
features of docker. Since then it has developed these features, if
|
|
||||||
Mesos/Marathon fails to produce results, then going back to kubernetes
|
|
||||||
is an option. However at the time of writing this Mesos/Marathon was
|
|
||||||
deemed to be more mature and stable.
|
|
||||||
|
|
||||||
Implementation
|
|
||||||
==============
|
|
||||||
|
|
||||||
Primary Assignee(s)
|
|
||||||
-----------
|
|
||||||
Angus Salkeld (asalkeld)
|
|
||||||
Kirill Proskurin (kproskurin)
|
|
||||||
Michal Rostecki (nihilifer)
|
|
||||||
|
|
||||||
Other contributor(s):
|
|
||||||
Harm Weites (harmw)
|
|
||||||
Jeff Peeler (jpeeler)
|
|
||||||
Michal Jastrzebski (inc0)
|
|
||||||
Sam Yaple (SamYaple)
|
|
||||||
Steven Dake (sdake)
|
|
||||||
<Please add your name here if you are getting involved in kolla-mesos>
|
|
||||||
|
|
||||||
Milestones
|
|
||||||
----------
|
|
||||||
|
|
||||||
Target Milestone for completion:
|
|
||||||
mitaka
|
|
||||||
|
|
||||||
Work Items
|
|
||||||
----------
|
|
||||||
1. Allow a custom startup script to run (change in Kolla)
|
|
||||||
2. Add startup scripts to kolla-mesos to read config from zookeeper
|
|
||||||
instead of bindmounted directory. Propose oslo.config changes to
|
|
||||||
use this method (oslo work done in parallel, initially this will be
|
|
||||||
done in the startup script).
|
|
||||||
3. Add startup scripts for service discovery so that services only
|
|
||||||
start once their needs are fulfilled.
|
|
||||||
a. register a service once a service is running
|
|
||||||
b. wait for dependent services if they are needed before starting
|
|
||||||
a service.
|
|
||||||
c. DNS and LB self-configuration based on service registry information
|
|
||||||
5. Add bootstrapping code to install Marathon, Zookeeper,
|
|
||||||
Mesos master and slave.
|
|
||||||
6. Add calls to to marathon to deploy containers.
|
|
||||||
7. Add support for kolla-mesos to kolla-cli.
|
|
||||||
|
|
||||||
Testing
|
|
||||||
=======
|
|
||||||
|
|
||||||
Functional tests will be implemented in the OpenStack check/gating system to
|
|
||||||
automatically check that the Mesos/Marathon deployment works for an AIO environment.
|
|
||||||
|
|
||||||
Documentation Impact
|
|
||||||
====================
|
|
||||||
A quick start guide will be written to explain how to deploy.
|
|
||||||
A develop guide will be written on how to contribute and how the deployment works.
|
|
||||||
|
|
||||||
References
|
|
||||||
==========
|
|
||||||
|
|
||||||
- [1] https://mesosphere.github.io/marathon/docs/constraints.html
|
|
||||||
- [2] https://mesosphere.github.io/marathon/
|
|
||||||
- http://radar.oreilly.com/2015/10/swarm-v-fleet-v-kubernetes-v-mesos.html
|
|
||||||
- https://www.wehkamplabs.com/blog/2015/10/15/applying-consul-within-the-blaze-microservices-platform/
|
|
||||||
Loading…
Reference in New Issue
Block a user