![Ryan Hallisey](/assets/img/avatar_default.png)
An outline of the kolla-kubernetes architecture. Change-Id: I5338f08d8c224fe6f877116f8ec2054ce67088c2
392 lines
16 KiB
ReStructuredText
392 lines
16 KiB
ReStructuredText
=============================
|
|
Kolla-kubernetes Architecture
|
|
=============================
|
|
|
|
OpenStack deployment requires multiple sequential steps to occur in workflow
|
|
like fashion. This puts the onus on Kubernetes to handle dependency management
|
|
and task ordering for each OpenStack service, which it currently doesn't. Using
|
|
the kolla-kubernetes fencing pod [1] and etcd-operator [2] as an examples, the
|
|
community will evaluate adding Kubernetes Operators [3] as a means of handling
|
|
the deployment of OpenStack.
|
|
|
|
In addition, there will be an effort to use the Kubernetes native package
|
|
manager, the Helm project [4]. Kolla-kubernetes will evaluate using Helm charts
|
|
as a means for making each OpenStack service available in the native Kubernetes
|
|
ecosystem and increase consumability of Kolla containers in real world
|
|
deployments and across communities.
|
|
|
|
Problem description
|
|
===================
|
|
|
|
In order to execute the OpenStack day 1 and day 2 workflows (deploy, upgrade,
|
|
and reconfigure), the community needs a way to handle running a series of tasks
|
|
in a certain order. Kubernetes does not natively support workflow operations.
|
|
|
|
OpenStack as a whole is highly customizable and flexible. A deployment tool
|
|
needs to be equally customizable and flexiable in order for a user to get
|
|
the most benifit from it.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
1. The kolla-kubernetes community will write Kubernetes Operators in **Python** as
|
|
part of the 1.0 release and revisit writing them in **Go** at a later date. The
|
|
reason for this is because Go lacks the ecosystem required by the TC in order to
|
|
allow for it to be used as part of OpenStack. Specifically, it has to do with
|
|
tools and code distribution.
|
|
|
|
2. The kolla-kubernetes community will write Helm charts for each of the OpenStack
|
|
services.
|
|
|
|
3. The kolla-kubernetes community will implement Dependency Init Containers.
|
|
|
|
Dependencies
|
|
------------
|
|
|
|
- Helm >= v2.0
|
|
- Kubernetes >= 1.4
|
|
|
|
Dependency Init Container
|
|
=========================
|
|
|
|
The dependency init container is a Kubernetes Init container object. The init
|
|
container will be the first container run in a pod and its job check if its safe
|
|
for a service to run. This layer is great for solving low level dependency
|
|
issues that operators are overkill for doing.
|
|
|
|
A good example of this is openvswitch needing to wait until ovs-db is running
|
|
in order to proceed.
|
|
|
|
Dependency List for the Keystone Pod
|
|
------------------------------------
|
|
|
|
Init container for keystone pod would be blocked until these dependencies are
|
|
met:
|
|
- MariaDB pod is ready
|
|
- Keystone bootstrap job is done
|
|
- Keystone configmap is present
|
|
|
|
After these conditions are met, the init container is marked as ready and
|
|
keystone pod can be deployed.
|
|
|
|
Kubernetes Operator
|
|
===================
|
|
|
|
*Kubernetes Controller* - A Kubernetes Controller is a piece of code that
|
|
manages the lifecycle of a complex application [5].
|
|
*Kubernetes Operator* - A Kubernetes Operator is defined as a containerized
|
|
Kubernetes Controller that uses the Kubernetes ThirdPartyResource [6]. The
|
|
Kubernetes Operator is meant to be directed by input, configuration, and action
|
|
from the user.
|
|
|
|
The Kubernetes Operator for deploying Keystone will perform the following tasks:
|
|
1. Check Mariadb exists
|
|
2. Check if Keystone exists
|
|
3. Read Keystone configuration file from the Kubernetes ThirdPartyResource and
|
|
register it as a configmap
|
|
4. Perform any user directed actions from the Kubernetes ThirdPartyResource
|
|
5. Use Helm to run a Kubernetes Job to create the Keystone database
|
|
6. Use Helm to run a Kubernetes Job to create the users and roles
|
|
7. Execute any additional startup tasks to bring the service to a ready state
|
|
8. Use Helm to run Kubernetes pods for Keystone
|
|
9. The Dependency Init Container determines its safe to run Keystone
|
|
10. The Keystone pod starts
|
|
|
|
User perspective of a Kubernetes Operator
|
|
------------------------------------------
|
|
|
|
OpenStack lifecycle management is a workflow and Kubernetes Operators are the
|
|
implementation of it. However, using a kubernetes Operator needs to be optional
|
|
because a user may want to handle all the necessary lifecycle steps by hand.
|
|
Therefore, the Kubernetes Operator needs to be detached from the services
|
|
themselves so that it can be flexible.
|
|
|
|
Code Outline
|
|
------------
|
|
|
|
The Controller code will live in the kolla-kubernetes repo. The Kubernetes
|
|
Operator container(s) will be added as kolla image(s) and exist in the
|
|
kolla-kubernetes repo.
|
|
|
|
Layering
|
|
========
|
|
|
|
The kolla-kubernetes project is broken down into multiple consumable building
|
|
blocks. The building blocks are referred to as **layers**.
|
|
|
|
**layer** - A layer in kolla-kubernetes has the following properties:
|
|
1. Optional - it can be skipped or turned off by an operator
|
|
2. Performs a specific task
|
|
3. All layers can be used for deployment
|
|
|
|
The diagram below is a representation of option 1 outlined in the
|
|
Kolla-kubernetes Design Options section.
|
|
|
|
As an example, a user needs the ability to choose the layer that best fits their
|
|
needs and preferences. That could mean skipping all upper layers and using layer
|
|
2.1 for deployment, but using layer 4 for upgrades. Also, a user could jump in
|
|
at layer 5 for deployment, then use layer 3 for upgrades or other lifecycle
|
|
operations.
|
|
_____________________
|
|
| OpenStack |
|
|
| Kubernetes Operator | layer 5 - The Kubernetes Operator that will have
|
|
|_________ __________| oversight of all the other Kubernetes
|
|
|| Operators
|
|
_________\/__________
|
|
| Service |
|
|
| Kubernetes Operator | layer 4 - Executes Helm packages for a service.
|
|
|_________ __________|
|
|
||
|
|
_________\/__________
|
|
| |
|
|
| Helm Package | layer 3 - Organizes the Kubernetes resources for each
|
|
|_________ __________| pod
|
|
||
|
|
_________\/__________
|
|
| |
|
|
| Kubernetes resource | layer 2 - Deploys pods/deployments/replication-
|
|
| _________________ | controllers/daemon-sets for an OpenStack
|
|
| | Dependency Init | | service.
|
|
| | Container | |layer 2.1- The Dependency Init Container checks to be
|
|
| |_________________| | sure dependencies are met before starting
|
|
|_________ __________| the service.
|
|
||
|
|
_________\/__________
|
|
| |
|
|
| OpenStack Service | layer 1 - Kolla container
|
|
|_____________________|
|
|
|
|
5. OpenStack Kubernetes Operator (optional)
|
|
4. Service Kubernetes Operator (optional)
|
|
3. Helm package (optional)
|
|
2. Kubernetes service pod
|
|
2.1 Dependency Init Container (optional)
|
|
1. OpenStack service
|
|
|
|
Kolla-kubernetes Design
|
|
=======================
|
|
|
|
The goal of this section is to provide the best kolla-kubernetes design so that
|
|
OpenStack lifecycle management will abide by the following principles:
|
|
|
|
- Granular - Each resource is a well defined building block with a purpose
|
|
- Consumable - Any user or project is able to easily use kolla-kubernetes
|
|
- Flexible - The project is capable of being used in different ways
|
|
- Customizable - The project is capable of adopting new use cases
|
|
- Debuggable - It is clear when a resource is misbehaving and why
|
|
|
|
There were six models considered for how the community will deploy OpenStack on
|
|
Kubernetes. Option 1 is highlighted below as the option the community found
|
|
the most appealing:
|
|
|
|
**Kolla-kubernetes will be using option 1 for its design model**
|
|
Option 1:
|
|
There are multiple Kubernetes Operators. One Kubernetes Operator per each
|
|
**service** (nova, keystone, neutron). There is a single OpenStack Kubernetes
|
|
operator that orchestrates each **service** Kubernetes Operator. The Dependency
|
|
Init Container exists as a Kubernetes Init object within a **service** pod.
|
|
The Dependency Init Container sits at the pod level to handle simpler dependency
|
|
resolution that isn't required by Kubernetes Operators.
|
|
|
|
- This option provides a good amount of granularity in that there is a clear
|
|
separation between the different layers of abstraction. *Every layer is*
|
|
*optional and can be used for deployment*.
|
|
|
|
Other Options That Were Considered
|
|
----------------------------------
|
|
|
|
Option 2:
|
|
There are multiple Kubernetes Operators. One Kubernetes Operator per each
|
|
**service** (nova, keystone, neutron). There is a single OpenStack kubernetes
|
|
operator that orchestrates each **service** Kubernetes Operator.
|
|
|
|
- This option provides a good amount of granularity in that there is a clear
|
|
separation between the different layers of abstraction. This could have
|
|
more layers or it might be the right amount of abstraction a user is looking
|
|
for.
|
|
|
|
Option 3:
|
|
There is a single OpenStack Kubernetes Operator that handles the deployment of
|
|
OpenStack.
|
|
|
|
- Though the simplicity is appealing here, this option is concerning because
|
|
it does not reflect the complexity, consumability, or granularity OpenStack
|
|
lifecycle management requires.
|
|
|
|
Option 4:
|
|
There are multiple Kubernetes Operators. One Kubernetes Operator per each
|
|
**micro service** (nova-api, nova-conductor, nova-scheduler). Above that,
|
|
there is a Kubernetes Operator per **service** that controls the micro
|
|
service Kubernetes Operator. At the top level, there is a single OpenStack
|
|
Kubernetes Operator that orchestrates each **service** Kubernetes
|
|
Operator.
|
|
|
|
- This option provides a high amount of granularity with every service being
|
|
thinned out to the microservice level. This option will be the most costly
|
|
in terms of code written since every service will have multiple Kubernetes
|
|
Operators. Also, there needs to be an account of the additional layering
|
|
being more granular. More layers will add flexibility for the user.
|
|
|
|
Option 5:
|
|
There are multiple Kubernetes Operators. One Kubernetes Operator per each
|
|
**role** (compute, controller, networking, monitoring). There is a single
|
|
OpenStack Kubernetes Operator that orchestrates each **role** Kubernetes
|
|
Operator.
|
|
|
|
- This option provides an average amount of granularity. The roles would
|
|
have to be completely customizable and exposed to the kubernetes operator.
|
|
There isn't a defined layering to handle an individual lifecycle task for a
|
|
specific service of a role in this model.
|
|
|
|
Option 6:
|
|
There is no OpenStack Kubernetes Operator. Only Kubernetes Operators for
|
|
Mariadb, rabbitmq, memcached, and to handle operations like back-ups or
|
|
disaster recovery. Entrypoints are used for workflow management.
|
|
|
|
- This is a simpler approach which subjects the orchestration layer to
|
|
the pod level.
|
|
|
|
- There is a concern here for trying to go too far to achieve maximum
|
|
usability. In Kolla's past, this has shown to create a lack of
|
|
complexity and flexibility which are both required for an OpenStack
|
|
deployment.
|
|
|
|
- OpenStack is a complex application where day 2 operations demand a lot of
|
|
care. If we handle complex operations at layer 1 (in the diagram), the
|
|
containers will have to each carry all logic required to perform all
|
|
operations at run time. Therefore, a user can easily run into a situation
|
|
where the cluster is expected to do something, but the code underneath does
|
|
something else entirely.
|
|
|
|
- Debugging is a huge pain in layer 1 because it is a challenge to know where
|
|
the workflow failed and how it failed with the logic scattered across all
|
|
the containers.
|
|
|
|
- The visibility and consumability are difficult because like Kolla learned,
|
|
deploying logic from the client side is more effective then having logic
|
|
run server side. Operators are the client side controllers, while
|
|
entrypoints are run in the containers (server side). Kolla has shown
|
|
that client side logic is far more effective for deploying a complex tool
|
|
like OpenStack because a user has more control over what is happening.
|
|
|
|
Workflow Example
|
|
================
|
|
|
|
As an example, if a user had a cluster with Keystone and MariaDB already running
|
|
and wanted to run Glance, this is what would occur at each layer:
|
|
|
|
User - The user creates a custom config file for glance-api and saves it in
|
|
/etc/kolla/glance-api/config.json.
|
|
|
|
Next, the user runs the OpenStack Kubernetes Operator and inputs the
|
|
customized glance-api config file as part of the Kubernetes
|
|
ThirdPartyResource. Glance-registry will use the default config file
|
|
in this example.
|
|
|
|
Layer 5 - The OpenStack Kubernetes Operator will look at the config files and
|
|
merge any changes overwriting the existing config files. Then, the
|
|
Kubernetes Operator looks at what the user requested for deployment.
|
|
|
|
The Kubernetes Operator checks if the cluster has deployed the
|
|
service then resolves the dependencies required for the service to be
|
|
deployed.
|
|
|
|
The OpenStack Kubernetes Operator spawns the Glance Kubernetes
|
|
Operator.
|
|
|
|
Layer 4 - The Glance Kubernetes Operator will gather the config data placed in
|
|
its Kubernetes ThirdPartyResource and create ConfigMaps for the config
|
|
files.
|
|
|
|
The Glance Kubernetes Operator will use Helm to start a Kubernetes Job
|
|
which creates the Glance DB user and password. Also, use Helm to run
|
|
the Kubernetes Job that will create the Keystone user and password.
|
|
|
|
Finally, the Glance Kubernetes Operator will run the Glance services
|
|
using Helm.
|
|
|
|
Layer 3 - The Glance Helm package will run the glance-api and glance-registry
|
|
pods based on the Glance kolla-kubernetes templates.
|
|
|
|
Layer 2 - The Kubernetes pods run glance-api and glance-registry as Deployments.
|
|
The templates will map the configmaps to the location Kolla expects
|
|
config files to appear.
|
|
- The Dependency Init Container checks to be sure glance-api and
|
|
glance-registry are safe to run. Once they are, the services start.
|
|
|
|
Layer 1 - The Kolla containers run. They each pick up the mounted config files
|
|
and run their service.
|
|
|
|
Helm
|
|
====
|
|
|
|
Helm inserts kolla-kubernetes into the Kubernetes app distribution system. That
|
|
way, a Kubernetes Operator can search for and consume the pieces of OpenStack as
|
|
building blocks to assemble a real world deployment. The user experience is
|
|
better using the Kubernetes native distribution system. In addition, Helm
|
|
provides a templating engine to ensure the templates are flexible and to make
|
|
the templates easy to version.
|
|
|
|
OpenStack and Kubernetes are different communities. In order to grow the
|
|
interop between the two, it makes sense for OpenStack to be distrubuted using
|
|
the Kubernetes native package manager.
|
|
|
|
Code Outline
|
|
------------
|
|
|
|
There will be a Helm chart for each OpenStack service [7]. The Helm charts will
|
|
premier in the kolla-kubernetes repo. After reaching some stabilization, the
|
|
community can decide to publish the charts to the incubation directory in the
|
|
Kubernetes repo [8].
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Community discussion etherpads [9][10].
|
|
|
|
Primary Assignee(s)
|
|
-------------------
|
|
Ryan Hallisey (rhallisey)
|
|
Steven Dake (sdake)
|
|
Kevin Fox (kfox1111)
|
|
Pete Birley (portdirect)
|
|
Michal Jastrzebski (inc0)
|
|
Mark Giles (mgiles)
|
|
Takashi Sogabe (sogabe)
|
|
Steve Wilkerson (srwilkers)
|
|
Duong Ha-Quang (duonghq)
|
|
Serguei Bezverkhi (sbezverk)
|
|
Surya Prakash Singh (sp_)
|
|
|
|
kolla-kubernetes team
|
|
|
|
< add your name here >
|
|
|
|
Work Items
|
|
----------
|
|
1. Write Kubernetes Operators required to run OpenStack
|
|
2. Write scripts that will execute the Kubernetes Operator
|
|
3. Write Helm charts for OpenStack services
|
|
4. Adjust the CLI to work with Kubernetes Operators and Helm
|
|
|
|
<Please add new work items that are worth mentioning in the spec>
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
< more docs >
|
|
|
|
References
|
|
==========
|
|
|
|
- [1] - https://review.openstack.org/#/c/383922/
|
|
- [2] - https://github.com/coreos/etcd-operator
|
|
- [3] - https://coreos.com/blog/introducing-operators.html
|
|
- [4] - https://github.com/kubernetes/helm
|
|
- [5] - https://coreos.com/blog/introducing-the-etcd-operator.html
|
|
- [6] - https://github.com/coreos/etcd-operator/blob/master/doc/design/arch.png
|
|
- [7] - https://github.com/sapcc/openstack-helm
|
|
- [8] - https://github.com/kubernetes/charts/tree/master/incubator
|
|
- [9] - https://etherpad.openstack.org/p/161115-kolla-kubernetes-cn-discussion
|
|
- [10] - https://etherpad.openstack.org/p/operator-base-class
|