Integrate Kata Containers into StarlingX
This specification is to integrate Kata Containers into StarlingX. So that StarlingX could support container run with kata container runtime. Story: 2006145 Task: 36054 Change-Id: I2de786ff678581f48026dff072f6d9841a3f4e85 Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This commit is contained in:
Shuicheng Lin
parent
05932b89d9
commit
6cfc5ebd66
|
|
@ -0,0 +1,322 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
========================================
|
||||
Integrate Kata Containers into StarlingX
|
||||
========================================
|
||||
|
||||
Storyboard: https://storyboard.openstack.org/#!/story/2006145
|
||||
|
||||
This story integrates Kata Containers into StarlingX. So that StarlingX could
|
||||
support container run with kata container runtime.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
Kata Containers is an open source community working to build a secure
|
||||
container runtime with lightweight virtual machines that feel and perform
|
||||
like containers, but provide stronger workload isolation using hardware
|
||||
virtualization technology as a second layer of defense.[0]
|
||||
|
||||
This story try to integrate Kata Containers into StarlingX, to address
|
||||
customer's concern for the security of container.
|
||||
|
||||
Due to there is still performance gap between Kata Containers and conventional
|
||||
containers [1], and Kubernetes cluster system cannot completely run on Kata
|
||||
Containers [2], Kata Containers is not used in default unless explicitly
|
||||
requirement is found in the Kubernetes's pod configuration file. Or in other
|
||||
words, regardless Kata Containers is enabled or not by this story, containers
|
||||
currently we have in StarlingX will not use kata container runtime in default,
|
||||
unless extra change is made. The method to declare the requirement is shared in
|
||||
"Proposed change" section.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
With Kata Containers supported in StarlingX, System developers, testers,
|
||||
operators, administrators could choose container runtime to use when run
|
||||
a container image, based on their needs. Such as if there is security concern
|
||||
for the pod/container, it could be selected to run with kata container. For the
|
||||
default case, runc will be selected as default low-level container runtime,
|
||||
which is the default low-level runtime for Kubernetes.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
Kubernetes is the only container orchestration tool in StarlingX. To integrate
|
||||
Kata Containers into StarlingX, Kuberentes will be configured to support kata
|
||||
container runtime.
|
||||
|
||||
For Kubernetes, there are two methods to select kata container runtime.
|
||||
1. Use annotation. Add "io.kubernetes.cri.untrusted-workload" annotation in
|
||||
pod file.[3]
|
||||
2. Use Runtime Class.[4]
|
||||
1st method is simpler, but it is deprecated. 2nd method is recommended.
|
||||
Runtime Class is supported in Kubernetes since v1.14, and current Kubernetes
|
||||
version in StarlingX master is v1.15. So use Runtime Class to select kata
|
||||
container runtime is recommended.
|
||||
|
||||
There are 3 major container runtime in kubernetes: docker/cri-o/containerd.[5]
|
||||
Docker runtime in Kubernetes supports runc only, it cannot be configured to
|
||||
support kata runtime. So in order to use kata container runtime, Kubernetes
|
||||
has to switch to use containerd or cri-o.
|
||||
|
||||
Per Kata Containers [6]: "As of Kata Containers 1.5, using shimv2 with
|
||||
containerd 1.2.0 or above is the preferred way to run Kata Containers with
|
||||
Kubernetes (see the howto). The CRI-O will catch up soon." So I choose
|
||||
containerd as the new container runtime for Kubernetes to support Kata
|
||||
Containers.
|
||||
|
||||
So we have two solutions here now for Kata Containers support in Kubernetes.
|
||||
Solution 1:
|
||||
A global flag will be added in ansible playbook local override file
|
||||
(localhost.yml) to let deployer choose whether Kata Containers support is
|
||||
enabled or not in StarlingX.
|
||||
If flag is not set or set as False, Kata Containers support is disabled,
|
||||
Kubernetes uses docker as container runtime. The flow will be like below:
|
||||
kubelet -> docker -> runc -> container image
|
||||
If flag is set as True, Kata Containers support is enabled, Kubernetes uses
|
||||
containerd as container runtime. For container doesn't require kata container,
|
||||
the flow will be like below:
|
||||
kubelet -> containerd -> runc -> container image
|
||||
For container requires kata container, the follow will be like below:
|
||||
kubelet -> containerd -> kata container runtime -> container image
|
||||
Advantage:
|
||||
For user doesn't use Kata Containers, system will act the same as previous.
|
||||
Code implementation will not affect current design. So it could be implemented
|
||||
in master directly piece by piece.
|
||||
Disadvantage:
|
||||
User cannot enable Kata Containers support after deployment.
|
||||
Code implementation needs consider both docker and containerd case.
|
||||
|
||||
Solution 2:
|
||||
No choice for user, and Kubernetes uses containerd as the container runtime.
|
||||
For container doesn't require kata container, the flow will be like below:
|
||||
kubelet -> containerd -> runc -> container image
|
||||
For container requires kata container, the flow will be like below:
|
||||
kubelet -> containerd -> kata container runtime -> container image
|
||||
Advantage:
|
||||
User always could use kata container.
|
||||
Code implementation will be simpler since just need consider containerd case.
|
||||
Disadvantage:
|
||||
Docker tools cannot be used for container deployed by Kubernetes. User should
|
||||
use kubectl to operate container or new tool like crictl/ctr.
|
||||
There is a cutover from docker to containerd. Code need be fully verified
|
||||
before merge.
|
||||
|
||||
To keep code clean and easy for maintenance, Solution 2 will be choosen for
|
||||
this feature implementation.
|
||||
More details for Solution 2 implementation is shared in "Work Items" section.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
CRI-O should be another option to enable Kata Containers support in Kubernetes.
|
||||
Containerd is selected instead of CRI-O, because Kata Containers prefer it.[5]
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None. This story does not change any existing data models.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None. This story does not change any existing REST APIs.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
There is no sensitive data touch for this feature, except containerd need
|
||||
touch certificate/key for local secure registry "registry.local".
|
||||
Kata Containers could help enhance system's security by "Runs in a dedicated
|
||||
kernel, providing isolation of network, I/O and memory and can utilize
|
||||
hardware-enforced isolation with virtualization VT extensions." [0]
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
With this feature implemented, users are able to run containers with
|
||||
kata-runtime.
|
||||
The major change for this feature is Kubernetes switch to use containerd as
|
||||
container runtime instead of docker. This change is transparent if users access
|
||||
the container with kubectl. Users will find container created by Kubernetes
|
||||
cannot be accessd by docker CLI. Users need use other CLI like crictl.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
For containers doesn't use kata-runtime, the performance should be the same as
|
||||
previous.
|
||||
For containers use kata-runtime, some performance drop is expected. Per Kata
|
||||
Containers website:"Delivers consistent performance as standard Linux
|
||||
containers; increased isolation with the performance tax of standard virtual
|
||||
machines."[0]
|
||||
Also there is a report of "I/O performance of Kata Containers". [1]
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
There is no change for the StarlingX deploy process.
|
||||
In node installation stage, a few RPMs for Kata Containers will be installed
|
||||
automatically in each node in order to support Kata Containers.
|
||||
In node configuration stage, docker and Kubernetes will be auto configured to
|
||||
support kata container runtime.
|
||||
After deployment, users need to update application tarball file to select kata
|
||||
runtime if want to use kata-runtime. Current application tarball will still
|
||||
work with runc runtime, just the same as before.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
The major change for this feature is, Kubernetes will switch to use containerd
|
||||
as container runtime. So any code implemented as docker specific in Kubernetes
|
||||
need switch to use containerd.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
This feature will introduce several new rpms for kata container. Kubernetes may
|
||||
be initialized with another container runtime during upgrade. Per my test,
|
||||
container runtime selection is per node. It means it is independent for the
|
||||
container runtime for each node. So this feature implementation should not
|
||||
affect upgrade.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Shuicheng Lin
|
||||
Yi Wang
|
||||
|
||||
Repos Impacted
|
||||
--------------
|
||||
|
||||
* stx-integ - add containerd configuration files
|
||||
* stx-config - add containerd confguration
|
||||
* stx-ansible-playbook - configure Kubernetes to use containerd
|
||||
* stx-root - add kata container rpms to ISO
|
||||
* stx-metal - add containerd partition
|
||||
* stx-tools - add kata container repo and rpms
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
Add kata container
|
||||
* Add kata container rpms to ISO and install to nodes
|
||||
|
||||
There is repo for Kata Containers rpms on CentOS 7. [7]
|
||||
kata-runtime rpm will be added to ISO explicitly, and other rpms will be
|
||||
added as dependency automatically.
|
||||
Kata Containers uses qemu as the hypervisor. Kata also could be configured
|
||||
to use Firecracker, which is out of scope of this story, and could be
|
||||
handled later if there is request.
|
||||
|
||||
Support kata container in Kubernetes
|
||||
* Configure containerd/kubelet service
|
||||
|
||||
Containerd binary is already installed in StarlingX as part of docker.
|
||||
A package with containerd related configuration file will be added. File
|
||||
with parameter need be configured dynamically will be configured through
|
||||
puppet. File doesn't need be updated will be installed by package.
|
||||
|
||||
* Switch kubeadm to use containerd
|
||||
|
||||
This will be done in ansible playbook for controller-0, and puppet
|
||||
configuration for controller-1, both by kubeadm init cmd. For worker node,
|
||||
it will be configured to use containerd by kubeadm join cmd.
|
||||
|
||||
* Configure containerd/docker to share same partition for container storage
|
||||
|
||||
By default, docker and containerd has different path for container image
|
||||
storage. /var/lib/docker for docker, and /var/lib/containerd for containerd.
|
||||
This is already a 30G lvm partition(/var/run/docker) created for docker
|
||||
currently. This partition is mainly for save container images and container
|
||||
runtime ephemeral data. With current StarlingX system, there are two
|
||||
container images armada and tiller pulled by docker directly, and 2
|
||||
application platform-integ-apps and stx-openstack pulled by Kubernetes.
|
||||
Armada and tiller images consume 550 Mb disk, while 2 applications consume
|
||||
17G disk.
|
||||
To avoid create 1 more lvm partition for containerd. Containerd will be
|
||||
configured to use /var/lib/docker path also. And all container images will
|
||||
be pulled by containerd in default. But users are still able to pull and run
|
||||
container with docker.
|
||||
|
||||
* Add crictl as the CLI for containerd
|
||||
|
||||
crictl is a CLI for CRI-compatible container runtimes. It allows the CRI
|
||||
runtime developers to debug their runtime without needing to set up
|
||||
Kubernetes components.[9] It could help debug containerd related issues.
|
||||
There is a rpm for this tool from OpenSuSE repo with latest version, but it
|
||||
is configured to contact with cri-o in default, need update configure file to
|
||||
contact with containerd.
|
||||
|
||||
* Configure containerd to support insecure registry/local registry/proxy etc
|
||||
|
||||
This is to replace the docker specific function used in Kubernetes.
|
||||
Containerd also need be configured to support upper feature.
|
||||
Docker SDK is used in sysinv to pull container image, and push it to local
|
||||
registry. Containerd doesn't have similar python SDK. We need re-write the
|
||||
code to call containerd or CLI to achieve the same function.
|
||||
|
||||
Besides upper work items, there are other work items maybe need added during
|
||||
implementation of this story. Please check the story link in the beginning of
|
||||
this spec to get the full work item list.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
This specification depends upon the open source upstream:
|
||||
Kata Containers/containerd
|
||||
https://github.com/kata-containers/kata-containers
|
||||
https://github.com/containerd/containerd
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
This feature can be tested in a fully provisioned StarlingX system
|
||||
|
||||
* if kata container support flag is not set, or set as False, the system
|
||||
should perform as previous.
|
||||
* if kata container support flag is set as True, the system should also
|
||||
pass deploy/sanity test, and containers managed by Kubernetes should be
|
||||
managed by containerd also, and use runc as low-level runtime in default.
|
||||
With kata container required container, it should be run with kata runtime.
|
||||
|
||||
To check whether the running container is with kata runtime or not, we could
|
||||
login into the container, and use "uname" cmd to check the kernel version.
|
||||
Host's kernel version is 3.10 while kata's kernel version should be 4.x.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A new page should be added to show how to enable Kata Containers support, and
|
||||
how to explicitly require kata container runtime in Kubernetes. This feature
|
||||
will add a new container runtime containered for Kubernetes. Any docker
|
||||
specific operation in Kubernetes may be updated to include containerd
|
||||
operation.
|
||||
|
||||
References
|
||||
==========
|
||||
[0]: https://katacontainers.io/
|
||||
[1]: https://www.stackhpc.com/kata-io-1.html
|
||||
[2]: https://github.com/kata-containers/runtime/issues/1853
|
||||
[3]: https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-an-untrusted-pod-using-kata-containers
|
||||
[4]: https://kubernetes.io/docs/concepts/containers/runtime-class/
|
||||
[5]: https://kubernetes.io/docs/setup/production-environment/container-runtimes/
|
||||
[6]: https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-a-cri-implementation
|
||||
[7]: http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/CentOS_7/
|
||||
[8]: https://github.com/kata-containers/documentation/blob/master/install/docker/centos-docker-install.md
|
||||
[9]: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - stx-3.0
|
||||
- Introduced
|
||||
Loading…
Reference in New Issue