Spec for fuxi-kubernetes
Partial-Implements: blueprint fuxi-kubernetes Change-Id: I6fb1023b8911beca1a4ebe9cb48b0280d0254a52
This commit is contained in:
Hongbin Lu
parent
bd69d53d74
commit
c500f37432
BIN
doc/images/fuxi_k8s_components.png
Normal file
BIN
doc/images/fuxi_k8s_components.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
@ -24,6 +24,15 @@ Developer Docs
|
||||
|
||||
devref/index
|
||||
|
||||
Design Specs
|
||||
============
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
specs/pike/contrail_support
|
||||
specs/pike/fuxi_kubernetes
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
|
||||
176
doc/source/specs/pike/fuxi_kubernetes.rst
Normal file
176
doc/source/specs/pike/fuxi_kubernetes.rst
Normal file
@ -0,0 +1,176 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
===========================
|
||||
Fuxi Kubernetes Integration
|
||||
===========================
|
||||
|
||||
https://blueprints.launchpad.net/kuryr-kubernetes/+spec/fuxi-kubernetes
|
||||
|
||||
This spec proposes an approach to integrate Kubernetes with various OpenStack
|
||||
storage services, such as Cinder, Manila.
|
||||
|
||||
Kubernetes is a platform for automating deployment, scaling and operations of
|
||||
application containers across clusters of hosts. Kubernetes currently supports
|
||||
Cinder plugin that manages Cinder volumes and makes them available for pods.
|
||||
In addition, there are a number of third-party implementations of Kubernetes
|
||||
volume plugins (i.e. Flocker, REX-Ray) that allows Kubernetes to manage volumes
|
||||
backed by various cloud storage providers including OpenStack Cinder. All these
|
||||
solutions are mainly for overcloud use cases, in which pods are deployed on
|
||||
a set of cloud instances. In production, there is another class of use cases,
|
||||
that were referred as the undercloud use cases, in which Kubernetes are
|
||||
deployed as a control plane service and pods are running on servers other
|
||||
than cloud instances (i.e. baremetal servers).
|
||||
|
||||
This spec proposed a solution that will address both overcloud and undercloud
|
||||
use cases. The specific requirements are as follows:
|
||||
|
||||
- Integrate Kubernetes with various OpenStack storage services, such as Cinder
|
||||
and Manila.
|
||||
- Support various backends that are supported by the OpenStack storage
|
||||
services.
|
||||
- Support various volumes operations including but not limiting to provision,
|
||||
de-provision, attach, de-attach, mount, un-mount, resize, snapshot, backup
|
||||
and restore.
|
||||
- Support Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC)
|
||||
[1].
|
||||
- Whenever possible, reuse existing services or frameworks. For example,
|
||||
Kuryr-kubernetes has the framework to watch the Kubernetes API and handle
|
||||
changes of resources. Fuxi has implemented the logic of interfacing with
|
||||
various OpenStack storage services and their backends. This proposal
|
||||
suggested to reuse Kuryr-kubernetes and Fuxi instead of re-inventing the
|
||||
equivalent functionalities.
|
||||
- Pluggable architecture. Allows integrating with custom storage solutions
|
||||
via plugins.
|
||||
- Massively scalable. Support large scale Kubernetes deployment (i.e.
|
||||
hundreds of nodes) with massive workloads (i.e. thousands of containers).
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
There are several ways to integrate Kubernetes with OpenStack. If Kubernetes
|
||||
is hosted by the cloud servers provided by Nova, users can leverage the
|
||||
Kubernetes cloud provider feature to provision and attach Cinder volumes to
|
||||
hosts. However, if Kubernetes is hosted by servers other than Nova instances,
|
||||
there is no perfect solution that connects Kubernetes to various OpenStack
|
||||
storage services.
|
||||
|
||||
|
||||
Proposed Change
|
||||
===============
|
||||
In order to integrate Kubernetes with OpenStack and satisfy the requirements
|
||||
above, this spec proposes to develop two components: A volume provisioner and
|
||||
a FlexVolume driver. The volume provisioner runs on the Kubernetes control
|
||||
plane and is responsible to watch for Kubernetes API for changes of PVCs and
|
||||
provision PVs for PVCs. The FlexVolume driver will reside on each host that
|
||||
runs Kubelet and it will be called out by Kubelet to perform local operations,
|
||||
such as attach volumes to hosts, etc.. Both volume provisioner and FlexVolume
|
||||
driver will consume OpenStack storage services via Fuxi server.
|
||||
|
||||
.. image:: ../../../images/fuxi_k8s_components.png
|
||||
:alt: integration components
|
||||
:align: center
|
||||
:width: 100%
|
||||
|
||||
|
||||
Volume Provisioner
|
||||
------------------
|
||||
Volume provisioner is responsible for watching Kubernetes API for PVCs and
|
||||
make sure the corresponding storage assets (i.e. cinder volume) are
|
||||
provisioned, updated, or deleted in OpenStack. The volume provisioner will
|
||||
implement the 'ResourceEventHandler' interface of Kuryr-kubernetes for
|
||||
handling PVC events.
|
||||
|
||||
For each creation of PVC in k8s, the Kuryr-kubernetes's API watcher will
|
||||
trigger an event that will be eventually handled by volume provisioner.
|
||||
On receiving the event, the volume provisioner will provision the appropriate
|
||||
storage asset in OpenStack and create a PV in k8s to represent the provisioned
|
||||
storage asset. The volume provisioning workflow will be in compliance with
|
||||
the Kubernetes's out-of-tree provisioning specification [2]. The provisioned
|
||||
PV will be populated with necessary information for the volume driver to
|
||||
connect to the provisioned storage asset later.
|
||||
|
||||
The volume provisioner will call the REST API of fuxi server to do the actual
|
||||
provisioning, and fuxi server will in term provision storage assets by using a
|
||||
volume provider (i.e. cinder provider). Note that fuxi was originally designed
|
||||
to be a remote docker volume plugin, and this proposal proposes to reuse it
|
||||
for fuxi Kubernetes.
|
||||
|
||||
Similarly, for each update or deletion of PVC, the volume provisioner will
|
||||
call fuxi server to update or delete the corresponding storage assets at
|
||||
OpenStack and PVs at k8s.
|
||||
|
||||
|
||||
FlexVolume Driver
|
||||
-----------------
|
||||
FlexVolume [3] is a k8s volume plugin that allows vendor to write their own
|
||||
driver to support custom storage solutions. This spec proposes to implement
|
||||
a FlexVolume driver that enables Kubelet to consume the provisioned storage
|
||||
assets. The FlexVolume driver will implement the FlexVolume's driver interface
|
||||
that is consistent of a set of 'call-outs'.
|
||||
|
||||
After the PVs are provisioned by the volume provisioner, they will be picked by
|
||||
Kubelet and Kubelet will assign the PVs to a volume plugin based on its type.
|
||||
In our case, all PVs provisioned by our volume provisioner will be set to
|
||||
'flexVolume' type so FlexVolume will be invoked to handle these PVs.
|
||||
FlexVolume will parse the PVs to retrieve information and pass down those
|
||||
information to our FlexVolume driver via 'call-outs'.
|
||||
|
||||
Generally speaking, PVs will serve as medium for the volume provisioner to
|
||||
communicate with the FlexVolume driver. The volume provisioner is supposed
|
||||
to populate PVs with all the data that will be consumed by the FlexVolume
|
||||
driver later. For example, the volume provisioner might provision a Cinder
|
||||
volume and set the Cinder volume's name in a field of the created PV,
|
||||
so that the name can be passed down to the FlexVolume driver who will consume
|
||||
the Cinder volume.
|
||||
|
||||
The FlexVolume driver will leverage fuxi to do the actual processing (i.e.
|
||||
connect to the volume). The initial implementation will assume an instance of
|
||||
fuxi server is deployed to each host that run Kubelet/FlexVolume driver so that
|
||||
the fuxi server and the FlexVolume driver can communicate via localhost.
|
||||
In the second phrase, we will investigate the possibility to have a centralized
|
||||
fuxi server which both volume provisioner and FlexVolume driver will consume.
|
||||
This might require splitting fuxi into a server and a library. The library will
|
||||
be leveraged by the FlexVolume driver to perform local operations (i.e. volume
|
||||
bi-mounting) and the server will serve cluster-wide requests.
|
||||
|
||||
Note that FlexVolume has several known drawbacks. For example, it invokes
|
||||
drivers via shells, which requires executables pre-installed in the specified
|
||||
path. This deployment model doesn't work with operating systems like CoreOS
|
||||
in which the root file system is immutable. This proposal suggests to continue
|
||||
monitoring the evolution of k8s and switch to a better solution if there is
|
||||
one showed up.
|
||||
|
||||
|
||||
Alternatives
|
||||
============
|
||||
An alternative to FlexVolume driver is provide an implementation of k8s volume
|
||||
plugin. An obstacle of this approach is that k8s doesn't support out-of-tree
|
||||
volume plugin (beside using FlexVolume) right now. Therefore, the fuxi volume
|
||||
plugin needs to be reside in k8s tree and released with a different schedule
|
||||
from OpenStack.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Hongbin Lu
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
1. Implement a k8s volume provisioner.
|
||||
2. Implement a k8s FlexVolume driver.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
[1] https://kubernetes.io/docs/concepts/storage/persistent-volumes/
|
||||
[2] https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-provisioning.md
|
||||
[3] https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md
|
||||
Loading…
x
Reference in New Issue
Block a user