Recent commit 54f2f7d6c667e0d26211e713d0b1fd44a527cdaa made
changes to the install path of the containernetworking-plugins
from /usr/libexec/cni/ to /opt/bin/cni/ as part of making
/usr readonly to support OSTree.
Since the bond-cni plugin is not distributed with the other
containernetworking-plugins, the same change needs to be
made in the bond-cni package.
Closes-Bug: 1976111
Testing:
Ensure /opt/cni/bin/bond exists on both Debian and CentOS.
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I48b47100d14c77818daf42cb24b7146ae6672e35
This change adds the package k8s-cni-cache-cleanup to StarlingX's
Debian build
Test Plan:
PASS build Debian ISO
PASS install AIO-SX Debian ISO
PASS Check package k8s-cni-cache-cleanup is present
PASS Check presence of script /usr/local/sbin/k8s-cni-cache-cleanup
Story: 2009965
Task: 45461
Signed-off-by: Andre Fernando Zanella Kantek <AndreFernandoZanella.Kantek@windriver.com>
Change-Id: I277937ad1be326f75c3b5fc01a30e775a7b9ca0a
This commit adds the kubernetes plugin kubectl cert manager to the iso.
This is used to convert old v1alpha2 and v1alpha3 cert manager
resources to v1 during a system upgrade. The plugin is not required
for debian because there are no old cert manager resources to convert.
Test Cases:
PASS: Convert our default DC certificates and issuers using
kubectl cert manager
Change-Id: I59f1b0e4d5d6ece1ccef43fee1acacd7b7e44efd
Story: 2009837
Task: 45372
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
This change makes a correction in kubeadm.conf for k8s 1.21.8 on
Debian originally committed in
https://review.opendev.org/c/starlingx/integ/+/827384
/etc/sysconfig does not exist on Debian.
Kubelet service environment variables file location is /etc/default/
on StarlingX Debian.
Test Plan:
Package builds successfully
Closes-Bug: 1955608
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: Ic3f7f6a514088a3ccbd7f99c0433a8144e8d0ade
OSTree structure requires /usr to be readonly as OSTree's dracut
hook creates a read-only bind mount over /usr.
1. deploy validate_postgresql_connection.sh directly to
/usr/local/bin. It was copied to the location after
installation.
2. move /usr/local/etc/ldapscripts to /etc/ldapscripts, files
need writable.
3. move /usr/libexec/cni to /opt/cni/bin. Plugins are installed
at runtime.
TCs:
provision aio-dx centos with /usr mount to readonly fs.
unlocked host
provision aio-sx debian and unlocked host.
upgrade AIO-DX from 21.12
upgrade AIO-SX from 21.12
successfully apply cert-manager and nginx-ingress-controller
Story: 2009101
Task: 44314
Change-Id: I99231f3f7db3d2d8eaceba137e13dea650370f71
Signed-off-by: Bin Qian <bin.qian@windriver.com>
This changes Debian package name for k8s 1.21.8 from "kubernetes" to
"kubernetes-1.21.8".
Until https://review.opendev.org/c/starlingx/integ/+/831343
is merged, version 1.21.8 is the only packaged version of
kubernetes on StarlingX Debian. In future, multiple kubernetes
versions will be supported on most, if not all, StarlingX releases.
Currently, Debian build server uses the value of 'debname' parameter in
the meta_data.yaml as the package name.
'debname' is an optional parameter in the meta_data.yaml.
If not provided, it uses package dir name as the package name
(kubernetes-1.21.8 in this case), which follows the preferred format
('kubernetes-<version>') for naming different versions of kubernetes
packages distinctly.
Test Plan:
PASS: Package builds successfully
PASS: Image builds successfully.
Story: 2009830
Task: 44638
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I46f7d9307f4254597557bb8be81ef471dcc7d73d
This introduces k8s-container-cleanup script that will be called
when containerd.service is stopped. The script detects whether systemd
state is 'stopping' due to shutdown/reboot, then stops all running
containers before the service shuts down.
During shutdown/reboot, some containers are not receiving the
SIGTERM signal. This leads to unexpected behaviour such as
generating huge coredumps.
There is an upstream issue regarding this:
https://github.com/kubernetes/kubernetes/issues/107158
The problem seems to be systemd related but this commit
addresses the problem with a workaround.
This reverts commit f3c18b0f79e3b145d378474b24d861926dd61a13.
The k8s-container-cleanup script is moved from kubelet.service
to containerd.service. The ExecStopPost that calls this script
is removed, and replaced with ExecStop in containerd.service
to call the script (in config-files repo).
The k8s-container-cleanup script requires containerd is running
in order to use crictl utility. The shutdown of kubelet and
containerd have unpredictable timing, so the cleanup must be done
in containerd.
Test Plan: On AIO-SX
PASS: Verify k8s-container-cleanup logs to daemon.log during 'stopping.
PASS: Manual change containerd/kubelet shutdown timing and verify.
k8s-container-cleanup running to completion before containerd stopped.
PASS: Reboot and verify k8s-container-cleanup running to completion.
PASS: Lock/unlock and verify k8s-container-cleanup running to completion.
PASS: Manually run spellintian tool against k8s-container-cleanup.sh.
PASS: Manually run shellcheck tool against k8s-container-cleanup.sh.
PASS: Zuul tox bashate tool against k8s-container-cleanup.sh.
Partial-Bug: 1964111
Change-Id: Ic8a9e257f861ae218a8520205eced3eaa580dd20
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
This commit fixes an issue that was seen if golang 1.17
was chosen as the toolchain to build the CNI package.
The go 1.17.5 build complains that the following vendored
modules should be explicitly required in the go.mod file:
github.com/coreos/go-iptables v0.6.0
github.com/safchain/ethtool v0.0.0-20210803160452-9aa261dae9b1
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e
If these are added to the go.mod file, a further complaint
is given that it no longer matches the information in
vendor/modules.txt
The patch files were generated by running go mod tidy for
the go.mod file, and go mod vendor for the vendor/modules.txt.
Since the bond-cni uses go 1.17 in the go directive of its
go.mod file, this commit locks down on this version to attempt
to prevent other issues from arising from new or other golang
versions.
Testing:
- CentOS build
- Debian build
- Spot check of bond-cni functionality on CentOS
Closes-Bug: 1966728
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I14638165db48cda9b89dd666b0c8b7c0a4e8e380
This work is part of Debian integration effort.
This removes a bootstrap issue.
Docker registry service is managed by puppet.
Disable the service, otherwise it will start and listen on the same
port keystone will, thus preventing keystone from starting.
Test:
PASS: build-pkgs & build-image
PASS: bootstrap
Change-Id: Ia7a4a8525af022ebff607700c42812611f3043e8
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
To align with k8s 1.21.8 to 1.23.1,
Upgrade containerd from 1.4.6 to 1.4.11
Note:
The change pulls containerd v1.4.12 debian source package
from debian salsa. The patch 0001-revert-to-v1.4.11.patch
reverts commits between version v1.4.11 and v1.4.12.
Note that the patch has no conflicts with any of the
other patches in debian source package. So it is safe
to apply it after they get applied.
Also, it is not a strict requirement to use 1.4.11
over 1.4.12. This is just to keep in-sync with the CentOS version
of StarlingX.
Test Plan:
containerd package builds successfully
All packages build successfully
Image builds successfully
Story: 2009845
Task: 44456
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I25a15a8cac1b324411b74b9f772978270d48a664
To align with k8s 1.21.8 to 1.23.1
Upgrade runc from version 1.0.0-rc95 to 1.0.2
Dependencies are packaged in this change.
Dependencies:
golang-github-checkpoint-restore-go-criu-dev (>= 5.1.0~)
golang-github-coreos-go-systemd-dev (>= 22.3.2~)
golang-github-opencontainers-specs-dev (>= 1.0.2.66~)
golang-github-cilium-ebpf-dev (>= 0.6.2~)
Note:
As of this date, except golang-github-cilium-ebpf-dev,
all other dependencies are new and not available in bullseye
main. They are available in sid though. It was preferred to
package them from source than adding them in base-bullseye.lst.
Also, runc failed to build if newest version (0.7.0) of
golang-github-cilium-ebpf-dev is used. The exact reason is not
clear. So it was preferable to package its minimum required
version.
Test Plan:
runc package builds successfully
All packages build successfully
Image builds successfully
Story: 2009845
Task: 44456
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I4139f9eb689a9e8c8e18c7a9b15fd2d592752ee5
This commit is a follow-on for d900a5b which introduced
the bond-cni plugin for Centos. In this commit we
introduce the same plugin for Debian.
Since there is no existing Debian package for the bond-cni,
we build it from source (tar.gz) to create the package.
The plugin is installed at /usr/libexec/cni/ to align
with where k8s expects plugins to be in StarlingX.
Testing:
Pass: Build
Pass: bond plugin present at /usr/libexec/cni/
Story: 2009800
Task: 44845
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: Ib9e805d587604f9b0a43a685b9b6970e5be1deb4
Testing:
* Deployed ISO with changes.
* Configured kube-cpu-mgr-policy=static.
* Verified that metrics-server were running on platform CPUs.
Partial-Bug: 1964503
Signed-off-by: Thiago Miranda <ThiagoOliveira.Miranda@windriver.com>
Change-Id: I447e4ccc113a4d0cc34a73bd71ac961305987c06
This commit uprevs the containernetworking-plugins to 1.0.1 on
Debian.
The version of this package has a Build-Depends on:
golang-github-appc-cni-dev >= 1.0.1
golang-github-vishvananda-netlink-dev >= 1.1.0.125
Currently, the Debian Bullseye release provides:
golang-github-appc-cni-dev = 0.8.1
golang-github-vishvananda-netlink-dev = 1.1.0-2
So in order to build the containernetworking-plugins 1.0.1, this
commit pulls in the appropriate versions of the dependencies
and builds them too.
The other thing to note in this commit is that the plugin install
path has been changed from /usr/lib/cni/ to /usr/libexec/cni/.
This aligns with where k8s expects to find CNI plugins on StarlingX
Testing:
- PASS: downloader -s
- PASS: full build
- PASS: install and ensure the plugins are present
at /usr/libexec/cni/
- PASS: install and ensure the package is the correct version
Story: 2009832
Task: 44635
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: Ic1c9a0d7eb2adb831c6316e1ab72d288ac805929
Simple port of two existing patches without modification to
kubernetes 1.23.1. This enables the feature to make kubelet
isolcpus allocation SMT aware.
Depends-On: https://review.opendev.org/c/starlingx/compile/+/825651
Depends-On: https://review.opendev.org/c/starlingx/integ/+/825654
Story: 2008760
Task: 44190
Test Plan:
PASS: Launch isolcpus pod and verify new kubelet logs on target host
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I2c18dea1b1f9a8a1c5e183e104a832ac872764e6
Qemu and kubernetes build failed to build under a 3 GB ramdisk.
The 6 GB ramdisk was ok.
Increase the minimum resources for these packages to 6 GB.
Closes-bug: 1964980
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I6027a4c02c15ef87a405ad300e967499c103b452
Changes for adding Kubernetes 1.23.1 in
StarlingX, including build environment updates.
The package builds successfully.
Built and installed an iso with K8s 1.23.1 on
AIO-SX.
Depends-On: https://review.opendev.org/c/starlingx/compile/+/825651
Story: 2009830
Task: 44424
Change-Id: I3e2b793d7b88057fc597b2445bddd137bb2b4fcf
Signed-off-by: Gleb Aronsky <gleb.aronsky@windriver.com>
Testing:
* Deployed ISO with changes.
* Configured kube-cpu-mgr-policy=static.
* Verified that metrics-server were running on platform CPUs.
Partial-Bug: 1964503
Signed-off-by: Thiago Miranda <ThiagoOliveira.Miranda@windriver.com>
Change-Id: I9bebe9ec27fcd70e89a4cae52bfacde993f958eb
When executing a reboot/shutdown
k8s pods are not receiving the SIGTERM
signal which leads some of them to
unexpected behaviour such as generating
huge coredumps.
There is an upstream issue regarding this:
https://github.com/kubernetes/kubernetes/issues/107158
The problem seems to be systemd related
but this commit addresses the problem
with a workaround.
This commit introduces a new script that
will cleanup all the remaing pods and will
be run after kubelet is stopped.
The script is executed successfully when
kubelet stops and the pods are stopped
before the system shuts down.
Closes-bug: 1964111
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
Change-Id: Ia0376aa510dd0dc3983e16cd89840726c15d6c92
Simple port of two existing patches without modification to
kubernetes 1.22.5. This enables the feature to make kubelet
isolcpus allocation SMT aware.
Story: 2008760
Task: 44190
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I6fb17f62d90bd6aa26b88dc7057ed7f667cf81b7
helm-toolkit's resources need to be updated
to use the new apiVersion and changes
proposed in:
https://kubernetes.io/docs/reference/using-api/deprecation-guide
This commit addresses just the changes needed for running
the armada chart.
It was deployed successfully on k8s 1.22.5.
Story: 2009888
Task: 44649
Change-Id: If38f62d178412f8a0974ec0af8ff2475800876be
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
This commit bumps the containernetwork-plugins package to v1.0.1.
This is part of the ongoing activity of refreshing kubernetes
components. In this case, various bug fixes throughout the
plugins have been picked up.
Test Plan:
- Successful build of the new package.
- Plugin functional testing:
main:
bridge
macvlan
ipvlan
vlan
ptp
host-device
meta:
sbr
vrf
tuning
portmap
bandwidth
ipam:
dhcp
static
host-local
Story: 2009832
Task: 44426
Change-Id: Id636b959cf500009b4f14bc10379a5edf969d032
Signed-off-by: Steven Webster <steven.webster@windriver.com>
To align with kubernetes 1.21.8,
Upgrade containerd from version 1.4.6 to 1.4.11
Upgrade runc from version 1.0.0-rc95 to 1.0.2
We continue to use no_btrfs build flag for containerd
as we do not use btrfs
Test Plan:
Built an iso (CentOS) and installed on AIO-DX lab
PASS: Run basic docker, ctr, crictl and runc commands
to create, list containers, images
PASS: Create new pods and PVCs and delete them
PASS: Lock-unlock and reboot hosts.
Check all pods are up back
Story: 2009845
Task: 44456
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I8e5ce0fd316e2e7f059c8abe5050732192f502a1
The new minimum supported k8s version
will be 1.21. This commit cleans the pkg
files needed to build the old k8s versions.
The pkgs build successfully. Deployed on
AIO-SX and AIO-DX, the k8s services were running ok.
Story: 2009859
Task: 44498
Change-Id: Ib39e9d1522a49c5788240781c8edee2bdffbc97a
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
By removing:
User=docker-registry
the service now runs as user root which allows ansible to use the
registry without hitting permissions issues.
Test Plan:
Pass: Execute ansible playbook in Debian OS
Pass: Service is active (running) after ansible playbook
Story: 2009101
Task: 44419
Signed-off-by: Fabricio Henrique Ramos <fabriciohenrique.ramos@windriver.com>
Change-Id: I06f3b0ed19d60400630bd01d3ae115fe44b6582b
The script will run everytime before the kubelet service is started.
It reads the reserved-cpus list for the kubelet from the service
environment file and sanitizes it on the basis of online CPUs.
If none of the reserved cpus is online, it removes the
--reserved-cpus flag from the environment file which allows
the kubelet to choose CPUs itself.
Sanitizing the reserved-cpus list everytime before the kubelet starts
assures that the kubelet will not fail to start due to unavailability
of one or more CPUs in the list.
By enabling or disabling CPU hyperthreading, available CPUs change.
This change will make sure changing CPU hyperthreading setting will
not lead to kubelet start failure after the system boots up.
Test Plan: (On AIO-SX)
PASS:
Initial Hyperthreading state: enabled
Host-lock->Reboot->Disable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Initial Hyperthreading state: disabled
Host-lock->Reboot->Enable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Manually restart the Kubelet service.
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Host-lock->Host unlock (without any config change).
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Packages built successfully on both Debian and CentOS.
Closes-Bug: 1955608
Change-Id: I699c5c36a56a50d4c48faa816edad69c17058079
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
The existing device manager code returns CPUs as devices in unsorted
order. This numerically sorts isolcpus allocations when SMT/HT is
enabled on the host. This logs SMT pairs, singletons, and algorithm
order details to make the algorithm understandable.
Example log for a 3 cpu isolcpus request:
2022-02-11T16:27:50.345 controller-0 kubelet[1531574]: info I0211
16:27:50.345529 1531574 manager.go:741] order_devices_by_sibling:
needed=3, smtpairs=[4 5 6 7 10 11], singletons=[8 12],
order=[8 4 5 6 7 10 11 12]
The specific host with SMT enabled has this topology:
LOGICAL CPU TOPOLOGY:
cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
core_id : 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7
thread_id : 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Before cpu allocation, host has Isolated_free: 4-8,10-12.
New pod gets the following isolcpus cpuset: 4-5,8.
Test Plan: (On AIO-SX, SMT enabled)
PASS: Verify cpu sort order for even needed and no singletons
PASS: Verify cpu sort order for odd needed and no singletons
PASS: Verify cpu sort order for even needed and singletons
PASS: Verify cpu sort order for odd needed and singletons
Story: 2008760
Task: 44190
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I1d743f80925b35ecee7936c12b0f4328f83b7eb2
