10 KiB
StarlingX: Move to Kernel v5.10
Move to Kernel v5.10 - Move to kernel v5.10 using source from the Yocto Project
Storyboard: https://storyboard.openstack.org/#!/story/2008921
With CentOS moving to rolling releases and the StarlingX community's commitment to a move to a Debian based userspace, the use of the CentOS kernel should not continue into future StarlingX releases. In this spec we propose moving to kernel version 5.10 LTS with source code provided by the Yocto Project's linux-yocto kernel. This move is proposed for any upcoming StarlingX releases, those based on CentOS userspace as well as for the kernel for the Debian based userspace.
Problem description
While StarlingX was based on CentOS it was worthwhile using the CentOS kernel source code with a limited set of StarlingX specific patches applied. StarlingX enjoyed many benefits with this approach, back-ports, bug reports, bug fixes and a preempt-rt implementation, to name a few. Despite the loss of git commit history1 it was the right thing to do and it worked well over several StarlingX releases.
After it was announced that CentOS would move to a rolling release2 the StarlingX community committed to move to a Debian based userspace. Continued use of the CentOS kernel is no longer optimal in this scenario as most of the benefits become moot and the biggest negative, the lack of git history becomes more acute. With StarlingX gaining users and deployments pushing hardware and software to their limits, our ability to review the kernel git history at the commit level is a must.
According to the CentOS stream announcement3, updates of CentOS 8 will terminate at the end of 2021. With STX 6.0 scheduled to be released in mid December 20214 this would leave the release containing a kernel with an uncertain future, just weeks after the STX 6.0 release.
Lastly there has been requests for kernel features not available or incomplete in the existing StarlingX kernel, such as virtual routing and forwarding (VRF). By moving kernels in the STX 6.0 release the availability of these features can be planned for immediately and are not dependent on the Debian transition work to complete.
Use Cases
By design we are planning to not affect any use cases. As with the move from the CentOS 7 kernel (v3.10) to the CentOS 8 kernel (v4.18) we intend to exploit the kernel's relatively stable application binary interface (ABI) to avoid any changes to StarlingX use cases.
Developers responsible for maintaining, debugging, configuring and tuning the StarlingX kernel will have to adjust their workflows to use Yocto Project resources instead of CentOS and Red Hat resources.
Similarly Deployers who actively do 'surveillance audits' on CVEs will have to make use of Yocto Project resources to check on CVE status and applicability instead of CentOS resources, to assist in their decision making around update schedule for instance.
The Release Team will have to review procedures which make use of CESA to ensure CentOS kernel related CVEs are not included. Yocto Project CVEs applicable to the kernel will have to be included in their place.
Despite a fairly stable ABI, changing kernel versions will bring with it other changes, for instance changes to the kernel command line, procfs, sysfs and devfs which may require updates to the installer, Ansible playbooks and other configuration infrastructure, these will mostly affect developers in these respective areas and will be transparent to end users.
Proposed change
This spec proposes:
- To move to kernel version 5.10 LTS in time for the STX 6.0 release
- To use the kernel source code from the Yocto Project's linux-yocto kernel
- To write packaging meta-data (spec files, etc.) to package the kernel in the same way as the CentOS kernel would be found, in the case of STX 6.0
- To write packaging meta-data (Debian dsc file, etc.) to package the kernel in the same way as the Debian kernel would be found, for the Debian transition
- To apply all applicable StarlingX kernel patches to the kernel prior to building
- To make available all existing out-of-tree kernel modules, keeping existing versions when possible or uprev'ing as required.
- Deal with any toolchain version issues using standard approaches, such as devtoolset-8.
Alternatives
We could delay a kernel version update and a move away from the CentOS kernel until we do the Debian transition. Unfortunately this delays the ability of StarlingX users to make use of newer kernel functionality. This also means more time spent without access to full git history which has hampered difficult kernel defect investigations. Most importantly this would have the STX 6.0 release completing just weeks before CentOS 8 stops getting updates at the end of 2021.
We could derive the kernel source from the Debian kernel instead of the Yocto Project kernel. The Yocto Project kernel, especially the preempt-rt kernel type, has proven to be a well maintained and stable kernel. As a Linux Foundation collaboration the Yocto Project is open to code submissions, and with several organizations involved in both the StarlingX and Yocto Project technical steering committees (TSCs) it should allow for easier integration over time.
We could use kernel source directly from mainline or another Linux distribution. Kernel source is only a small part of building and maintaining a kernel. The vast majority of all Linux distributions only apply patches amounting to hundreds of thousands of lines of code, representing less than 0.5% of all kernel code. Thus with greater than 99.5% shared code the real differences between using kernel source code from one source vs another are derived from compile time configuration, developer workflow, patching, CVE and bug tracking and mitigation, ... Again it is known that several organizations involved in StarlingX are also involved in the Yocto Project thus sharing kernel source code between these two projects has many advantages over using other sources of kernel source code.
Data model impact
None
REST API impact
None
Security impact
Standard kernel version uprev implications exist. The kernel does contain cryptographic algorithms which may have undergone updates. Userspace applications should not see any direct impact.
Other end user impact
None
Performance Impact
The Linux kernel is always undergoing optimizations and updates along with additions and bug fixes from version to version. Without a doubt these changes affect kernel and userspace performance, sometimes positively and sometimes negatively. Overall kernel performance does remain fairly constant. Having already done extensive benchmarks on kernel version 5.10 with the Yocto Project, overall performance remains on par with other kernel releases. Specific StarlingX use cases may be found to have a change in performance with the new kernel but these changes are anticipated to be negligible.
Other deployer impact
None
Developer impact
Developers who never touch the kernel will continue to never touch the kernel. Developers who are involved with kernel tuning, kernel debugging or kernel patching, or who work with any of the out of tree kernel modules, will not see major changes in their workflows for STX 6.0. The biggest change will be access to full git commit history and the ability to work on the kernel outside of the StarlingX context if they desire, ie. using Yocto Project workflows.
After the Debian transition kernel developers will have to adjust to new kernel patching procedures as we move from CentOS RPM based packages to DEB packages. These impacts will exist regardless of the acceptance of this spec.
Upgrade impact
None
Implementation
Assignee(s)
- Primary assignee:
-
Mark Asselstine - markawr
Other contributors:
- Vefa Bicakci - vbicakci
- Jiping Ma - jma1
- Li Zhou - lzhou2
Repos Impacted
starlingx/kernel starlingx/ansible-playbooks starlingx/update
Work Items
See storyboard. Additional items will be added after spec approval.
Dependencies
Relates to StarlingX Userspace transition to Debian (approved) https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_debian_transition.html
Adds a new dependency to the Yocto Project https://www.yoctoproject.org
Testing
All existing StarlingX runtime tests and use cases will be used to validate the new kernel. Testing will span as many different hardware systems as possible. Where possible kernel tests used by the Yocto Project will be ported to StarlingX and run allowing for testing the kernel directly as well as exercising the userspace/kernel interfaces.
Testing efforts will attempt to cover as many use cases as possible, across all image types, and cover both the standard and low latency kernels. Testing will include running benchmarks, ensuring the new kernel meets documented performance requirements. Where no performance requirements exists efforts will be made to run comparison benchmarks using the old and new kernels to ensure comparable performance.
Documentation Impact
Documents targeted at kernel developers will have to be reviewed for any CentOS specific content and updated accordingly. Most other documents will remain unchanged other than swapping out kernel version numbers.
References
History
| Release Name | Description |
|---|---|
| STX-6.0 | Introduced |