diff --git a/doc/images/fuxi_k8s_components.png b/doc/images/fuxi_k8s_components.png new file mode 100644 index 000000000..a30a5e23f Binary files /dev/null and b/doc/images/fuxi_k8s_components.png differ diff --git a/doc/source/index.rst b/doc/source/index.rst index a8529cc7e..4a99326f0 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -24,6 +24,15 @@ Developer Docs devref/index +Design Specs +============ + +.. toctree:: + :maxdepth: 1 + + specs/pike/contrail_support + specs/pike/fuxi_kubernetes + Indices and tables ================== diff --git a/doc/source/specs/pike/fuxi_kubernetes.rst b/doc/source/specs/pike/fuxi_kubernetes.rst new file mode 100644 index 000000000..ffa85f29b --- /dev/null +++ b/doc/source/specs/pike/fuxi_kubernetes.rst @@ -0,0 +1,176 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=========================== +Fuxi Kubernetes Integration +=========================== + +https://blueprints.launchpad.net/kuryr-kubernetes/+spec/fuxi-kubernetes + +This spec proposes an approach to integrate Kubernetes with various OpenStack +storage services, such as Cinder, Manila. + +Kubernetes is a platform for automating deployment, scaling and operations of +application containers across clusters of hosts. Kubernetes currently supports +Cinder plugin that manages Cinder volumes and makes them available for pods. +In addition, there are a number of third-party implementations of Kubernetes +volume plugins (i.e. Flocker, REX-Ray) that allows Kubernetes to manage volumes +backed by various cloud storage providers including OpenStack Cinder. All these +solutions are mainly for overcloud use cases, in which pods are deployed on +a set of cloud instances. In production, there is another class of use cases, +that were referred as the undercloud use cases, in which Kubernetes are +deployed as a control plane service and pods are running on servers other +than cloud instances (i.e. baremetal servers). + +This spec proposed a solution that will address both overcloud and undercloud +use cases. The specific requirements are as follows: + +- Integrate Kubernetes with various OpenStack storage services, such as Cinder + and Manila. +- Support various backends that are supported by the OpenStack storage + services. +- Support various volumes operations including but not limiting to provision, + de-provision, attach, de-attach, mount, un-mount, resize, snapshot, backup + and restore. +- Support Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) + [1]. +- Whenever possible, reuse existing services or frameworks. For example, + Kuryr-kubernetes has the framework to watch the Kubernetes API and handle + changes of resources. Fuxi has implemented the logic of interfacing with + various OpenStack storage services and their backends. This proposal + suggested to reuse Kuryr-kubernetes and Fuxi instead of re-inventing the + equivalent functionalities. +- Pluggable architecture. Allows integrating with custom storage solutions + via plugins. +- Massively scalable. Support large scale Kubernetes deployment (i.e. + hundreds of nodes) with massive workloads (i.e. thousands of containers). + +Problem Description +=================== +There are several ways to integrate Kubernetes with OpenStack. If Kubernetes +is hosted by the cloud servers provided by Nova, users can leverage the +Kubernetes cloud provider feature to provision and attach Cinder volumes to +hosts. However, if Kubernetes is hosted by servers other than Nova instances, +there is no perfect solution that connects Kubernetes to various OpenStack +storage services. + + +Proposed Change +=============== +In order to integrate Kubernetes with OpenStack and satisfy the requirements +above, this spec proposes to develop two components: A volume provisioner and +a FlexVolume driver. The volume provisioner runs on the Kubernetes control +plane and is responsible to watch for Kubernetes API for changes of PVCs and +provision PVs for PVCs. The FlexVolume driver will reside on each host that +runs Kubelet and it will be called out by Kubelet to perform local operations, +such as attach volumes to hosts, etc.. Both volume provisioner and FlexVolume +driver will consume OpenStack storage services via Fuxi server. + +.. image:: ../../../images/fuxi_k8s_components.png + :alt: integration components + :align: center + :width: 100% + + +Volume Provisioner +------------------ +Volume provisioner is responsible for watching Kubernetes API for PVCs and +make sure the corresponding storage assets (i.e. cinder volume) are +provisioned, updated, or deleted in OpenStack. The volume provisioner will +implement the 'ResourceEventHandler' interface of Kuryr-kubernetes for +handling PVC events. + +For each creation of PVC in k8s, the Kuryr-kubernetes's API watcher will +trigger an event that will be eventually handled by volume provisioner. +On receiving the event, the volume provisioner will provision the appropriate +storage asset in OpenStack and create a PV in k8s to represent the provisioned +storage asset. The volume provisioning workflow will be in compliance with +the Kubernetes's out-of-tree provisioning specification [2]. The provisioned +PV will be populated with necessary information for the volume driver to +connect to the provisioned storage asset later. + +The volume provisioner will call the REST API of fuxi server to do the actual +provisioning, and fuxi server will in term provision storage assets by using a +volume provider (i.e. cinder provider). Note that fuxi was originally designed +to be a remote docker volume plugin, and this proposal proposes to reuse it +for fuxi Kubernetes. + +Similarly, for each update or deletion of PVC, the volume provisioner will +call fuxi server to update or delete the corresponding storage assets at +OpenStack and PVs at k8s. + + +FlexVolume Driver +----------------- +FlexVolume [3] is a k8s volume plugin that allows vendor to write their own +driver to support custom storage solutions. This spec proposes to implement +a FlexVolume driver that enables Kubelet to consume the provisioned storage +assets. The FlexVolume driver will implement the FlexVolume's driver interface +that is consistent of a set of 'call-outs'. + +After the PVs are provisioned by the volume provisioner, they will be picked by +Kubelet and Kubelet will assign the PVs to a volume plugin based on its type. +In our case, all PVs provisioned by our volume provisioner will be set to +'flexVolume' type so FlexVolume will be invoked to handle these PVs. +FlexVolume will parse the PVs to retrieve information and pass down those +information to our FlexVolume driver via 'call-outs'. + +Generally speaking, PVs will serve as medium for the volume provisioner to +communicate with the FlexVolume driver. The volume provisioner is supposed +to populate PVs with all the data that will be consumed by the FlexVolume +driver later. For example, the volume provisioner might provision a Cinder +volume and set the Cinder volume's name in a field of the created PV, +so that the name can be passed down to the FlexVolume driver who will consume +the Cinder volume. + +The FlexVolume driver will leverage fuxi to do the actual processing (i.e. +connect to the volume). The initial implementation will assume an instance of +fuxi server is deployed to each host that run Kubelet/FlexVolume driver so that +the fuxi server and the FlexVolume driver can communicate via localhost. +In the second phrase, we will investigate the possibility to have a centralized +fuxi server which both volume provisioner and FlexVolume driver will consume. +This might require splitting fuxi into a server and a library. The library will +be leveraged by the FlexVolume driver to perform local operations (i.e. volume +bi-mounting) and the server will serve cluster-wide requests. + +Note that FlexVolume has several known drawbacks. For example, it invokes +drivers via shells, which requires executables pre-installed in the specified +path. This deployment model doesn't work with operating systems like CoreOS +in which the root file system is immutable. This proposal suggests to continue +monitoring the evolution of k8s and switch to a better solution if there is +one showed up. + + +Alternatives +============ +An alternative to FlexVolume driver is provide an implementation of k8s volume +plugin. An obstacle of this approach is that k8s doesn't support out-of-tree +volume plugin (beside using FlexVolume) right now. Therefore, the fuxi volume +plugin needs to be reside in k8s tree and released with a different schedule +from OpenStack. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: +Hongbin Lu + + +Work Items +---------- +1. Implement a k8s volume provisioner. +2. Implement a k8s FlexVolume driver. + + +References +========== +[1] https://kubernetes.io/docs/concepts/storage/persistent-volumes/ +[2] https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-provisioning.md +[3] https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md