This change a6a5349d02
(k8s-1.22.5: remove feature-gates)
adds a script that is run during upgrade activate. The script modifies
kubeadm cluster config and eventually updates kube-apiserver manifest
to remove deprecated features-gates in k8s 1.22.
As 'kubeadm init phase' is rerun in the script, it updates the
kube-apiserver manifest to be in sync with the kubeadm cluster config.
In that process, it nullifies the effect of these two commits,
04a1c1b080
(Rework advertise address in apiserver-change-param)
and 52ace69c83
(Amend kube-apiserver 1.23 configuration to use PSP)
This change adds a function to the script that preserves the effect
of above listed commits.
Test Plan:
On CentOS AIO-SX
PASS: Upgrade Successful. Check if advertise address in
kube-apiserver manifest before and after running
upgrade-activate is same.
Ensure that the seccomp profile configuration is
removed after upgrade-activate.
Kube-apiserver is running and cluster is accessible after
the upgrade.
PASS: No Shellcheck errors
Closes-Bug: 1986854
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: Ib97e14bc5b4ed208e65e16888e1380a3bd9fdb8f
The sanitize_kubelet_reserved_cpus.sh was being
installed as a directory rather than a script.
Test Plan:
Build/Bootstrap/Unlock on Debian AIO-SX
Verify that kubelet starts up, and the sanity file exists
Story: 2009964
Task: 45878
Signed-off-by: Al Bailey <al.bailey@windriver.com>
Change-Id: I966f8b13f7cbd65f1a3f015d75d628e38a166038
This commit backports IFNAME key support from v1.1.1 of the
k8s containernetworking-plugins:
c16cff9805
IFNAME key support allows one to use the keyword
'IFNAME' in a network attachment definition using the
tuning plugin. Without this, the actual interface name
(whether specified in the pod spec, or the default 'net<X>')
must be specified.
Example:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: hd0
spec:
config: '{
"cniVersion": "0.3.1",
"plugins": [
{
"name": "hd0",
"type": "host-device",
"device": "eth1000"
},
{
"type": "tuning",
"sysctl": {
"net.ipv6.conf.IFNAME.accept_ra": "0"
}
}
]
}'
The above example would disable the processing of
IPv6 router advertisements on the interface associated
with the network attachment definition, regardless of
what the interface has been named in the container.
Note: Currently, StarlingX supports v1.0.1 of the
containernetworking-plugins. Once the plugins have been
up-revved to v1.1.1, this patch can be removed.
Testing:
- Ensure patch is applied and build successful
on CentOS and Debian
- Perform a functional test of the tuning plugin using
the IFNAME key on CentOS and Debian
Story: 2010114
Task: 45693
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I4fc617390b25bcf74a2a319fcb4409a0633c4a31
This provides the Debian containerd package changes to include
k8s-container-cleanup script.
Test Plan: Debian:
PASS: Build containerd package
PASS: Build image
PASS: Install ISO for AIO-SX
PASS: Reboot host, verify we get daemon.log:
k8s-container-cleanup(283049): info : Stopping all containers.
Closes-Bug: 1964111
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I56170b98cf32c2e7e51b1c35779305a90cdc6db8
This script is intended to be run during platform upgrade.
('upgrade-activate' phase). It removes below feature gates
from kubeadm-config configmap and rewrites kube-apiserver
and kube-controller-manager manifests.
- SCTPSupport=true
- HugePageStorageMediumSize=true
- TTLAfterFinished=true
Background:
HugePageStorageMediumSize is deprecated in Kubernetes 1.22
SCTPSupport blocks kube-apiserver pod to spawn after control-plane upgrade
TTLAfterFinished value defaults to true from k8s 1.21
Test Plan (On CentOS)
On AIO-SX and AIO-DX:
PASS: Full platform upgrade successful.
Confirm kubeadm-configmap is updated, kube-apiserver
and controller-manager static manifests and processes
are updated with updated feature-gates after platform
upgrade.
PASS: Upgrade k8s 1.21.8 to 1.22.5 after platform upgrade.
Create a PV, PVC and deploy a pod.
PASS: Package builds successfully.
Story: 2009789
Task: 45627
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I51400c63226b532eed4a05fddb255b877cc5bbb5
This commit fixes an issue with the debian install of the
bond CNI plugin.
It was noticed that the 'bond' source directory was being
installed, rather than the built 'bond' binary.
Since the build output is a single binary, it will be
found in debian/bond-cni/, rather than the standard
debian/tmp/
dh_install is instructed where to find the build output
installed by dh_auto_install.
Testing:
- Debian build and install
- Confirmed the bond binary is at /var/opt/cni/bin/
- Functional testing to ensure a pod using the
bond plugin could be launched
Depends-On: https://review.opendev.org/c/starlingx/integ/+/844865
Closes-Bug: 1976111
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I2a504b8bfc210dc09487a496959da235dec82525
This provides the original Debian containerd package files:
rules, containerd.install. These files are contained within
the tarball: containerd-debian-1.4.12_ds1-1.tar.gz .
Subsequent changes to these are package customizations.
Test Plan: Debian
PASS: Build Debian containerd package
Partial-Bug: 1964111
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: Icf5356c94b64b2c786ee988ad34cdd0a6e25c915
Follow-up to https://review.opendev.org/c/starlingx/integ/+/845752.
As explained in the mentioned commit we need to build the armada
container image using the sources provided there.
Add support for that.
Partial-Bug: 1978409
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: I26e8117272a6925ea429be10be91ce20f6b71c8b
Currently, we use in the playbooks [1] an armada image built upstream
[2]. We use armada from upstream helm2 branch.
Armada needs to be patched to add support for k8s >=1.22.
Proposed an upstream patch, but we don't know how long it takes until
it is merged:
https://review.opendev.org/c/airship/armada/+/845392
Instead of waiting for upstream commit to merge, and an image be
generated, we provide the code change here, so an armada image with
k8s >=1.22 support can be generated. The k8s >=1.22 support is added
by patch 0003.
Necessary StarlingX build changes to support generating an container
image and push to https://hub.docker.com/r/starlingx will be
addressed in another commit.
Note: since we always used an upstream built armada container, this
package purpose was to provide helm chart overrides. We add a new
purpose: to release to opensource the changes we are about to do to
armada, since we'll be building a container image using these changes.
To achieve this we do the following:
- upversion sources from 7ef4b8643b5ec5216a8f6726841e156c0aa54a1a
to ddbdd7256c20f138737f6cbd772312f7a19f58b8. This ensures we are
patching the image used in the playbooks[2].
- create patches 0001 and 0002 to ensure there are no helm chart
changes between upversion. This reduces testing effort related to
original purpose (provide helm chart overrides) of this package.
- create patch 0003 to add k8s >=1.22 support.
- old patches are not changed, but renamed from 0001->0005 to
0004->0008 and regenerated.
Other notes:
We don't need to port this work to CentOS. This work is supposed to be
temporary until the upstream airship/armada commit merges.
Tests on Debian:
PASS: build-pkgs -c -p armada
PASS: make images
Upload image to controller, use it.
Using the new armada image do an apply,remove,apply,remove,apply
chain for a custom StarlingX app.
[1]: https://opendev.org/starlingx/ansible-playbooks/src/branch/
master/playbookconfig/src/playbooks/roles/common/
load-images-information/vars/k8s-v1.22.5/system-images.yml#L5
[2] quay.io/airshipit/armada:
ddbdd7256c20f138737f6cbd772312f7a19f58b8-ubuntu_bionic
Partial-Bug: 1978409
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: Id51c241a3965ef462d325da4ffce37a81693a9f4
This plugin is needed to ease migration of helmv2 release to helmv3,
therefore enabling migration of Armada apps to FluxCD
TEST PLAN:
PASS: 2to3 is installed to /usr/local/share/helm/plugins/2to3
Story: 2009138
Task: 45584
Depends-on: https://review.opendev.org/c/starlingx/tools/+/845273
Signed-off-by: Lucas Cavalcante <lucasmedeiros.cavalcante@windriver.com>
Change-Id: I83d572bf8903c0d6e4daf189e69487956b0f8bcc
This will allow bootstrap on Debian to work without unlocking ostree.
Currently /opt/cni is a symlink to a /usr subdir.
/usr is mounted read-only. This lead to issues when containers try to
write inside /opt/cni.
Update software to use /var/opt/cni instead.
The problematic symlink is created by the meta-lat component.
This commit can be reverted later if the meta-lat design is changed.
This is an enhancement to the following series:
https://review.opendev.org/c/starlingx/ansible-playbooks/+/825354https://review.opendev.org/c/starlingx/integ/+/825346https://review.opendev.org/c/starlingx/stx-puppet/+/825355https://review.opendev.org/c/starlingx/integ/+/843965
Test on AIO-SX:
CentOS:
PASS: unlocked enabled available
Debian:
PASS: bootstrap without ostree unlock.
PASS: unlocked enabled available
Story: 2009101
Task: 44314
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: Id6ced63f913ed21954c379b031ae74683cd4d86f
This change adds missing patches to debian armada-helm-toolkit
package which are already present for its CentOS equivalent.
These patches are particularly important because Kubernetes 1.22
deprecated below k8s apiversions that armada-helm-toolkit uses.
- 'extensions/v1beta1' for 'Ingress' kind
- 'rbac.authorization.k8s.io/v1beta1' for 'Role' and 'RoleBinding'
- 'extensions/v1beta1' for kind 'PodSecurityPolicy'
'Ingress' should now use apiversion 'networking.k8s.io/v1'.
'Role' and 'RoleBinding' should now use apiversion
'rbac.authorization.k8s.io/v1'.
'PodSecurityPolicy' should now use apiversion 'policy/v1beta1'.
Reference: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#migrate-to-non-deprecated-apis
Test Plan:
PASS: Package builds successfully
PASS: Image builds successfully
PASS: Armada helm chart gets installed successfully during ansible
bootstrap for k8s 1.23.1
Story: 2009888
Task: 44649
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: Ic2b6a982e53b01ec24f4ab1bcf61025c59acf86e
Containerd uses the "process" killmode which shutsdown
shim v1 container processes but it does not shutdown
v2 container processes. As a result, when shutting down
the server it will result in a longer shutdown time
than compared to Centos 7.
This is a temporary workaround until we update to containerd
1.5+.
Test Plan
PASS Build containerd with patch
PASS Boot and unlock server
PASS Reboot server check for continerd-shim processes
in the syslog after the server reboots
Story: 2009845
Task: 44456
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: Iac496e9f2b7f3ccded5ea3e034db8bac2cfc0125
Recent commit 54f2f7d6c667e0d26211e713d0b1fd44a527cdaa made
changes to the install path of the containernetworking-plugins
from /usr/libexec/cni/ to /opt/bin/cni/ as part of making
/usr readonly to support OSTree.
Since the bond-cni plugin is not distributed with the other
containernetworking-plugins, the same change needs to be
made in the bond-cni package.
Closes-Bug: 1976111
Testing:
Ensure /opt/cni/bin/bond exists on both Debian and CentOS.
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I48b47100d14c77818daf42cb24b7146ae6672e35
This change adds the package k8s-cni-cache-cleanup to StarlingX's
Debian build
Test Plan:
PASS build Debian ISO
PASS install AIO-SX Debian ISO
PASS Check package k8s-cni-cache-cleanup is present
PASS Check presence of script /usr/local/sbin/k8s-cni-cache-cleanup
Story: 2009965
Task: 45461
Signed-off-by: Andre Fernando Zanella Kantek <AndreFernandoZanella.Kantek@windriver.com>
Change-Id: I277937ad1be326f75c3b5fc01a30e775a7b9ca0a
This commit adds the kubernetes plugin kubectl cert manager to the iso.
This is used to convert old v1alpha2 and v1alpha3 cert manager
resources to v1 during a system upgrade. The plugin is not required
for debian because there are no old cert manager resources to convert.
Test Cases:
PASS: Convert our default DC certificates and issuers using
kubectl cert manager
Change-Id: I59f1b0e4d5d6ece1ccef43fee1acacd7b7e44efd
Story: 2009837
Task: 45372
Signed-off-by: Jerry Sun <jerry.sun@windriver.com>
This change makes a correction in kubeadm.conf for k8s 1.21.8 on
Debian originally committed in
https://review.opendev.org/c/starlingx/integ/+/827384
/etc/sysconfig does not exist on Debian.
Kubelet service environment variables file location is /etc/default/
on StarlingX Debian.
Test Plan:
Package builds successfully
Closes-Bug: 1955608
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: Ic3f7f6a514088a3ccbd7f99c0433a8144e8d0ade
OSTree structure requires /usr to be readonly as OSTree's dracut
hook creates a read-only bind mount over /usr.
1. deploy validate_postgresql_connection.sh directly to
/usr/local/bin. It was copied to the location after
installation.
2. move /usr/local/etc/ldapscripts to /etc/ldapscripts, files
need writable.
3. move /usr/libexec/cni to /opt/cni/bin. Plugins are installed
at runtime.
TCs:
provision aio-dx centos with /usr mount to readonly fs.
unlocked host
provision aio-sx debian and unlocked host.
upgrade AIO-DX from 21.12
upgrade AIO-SX from 21.12
successfully apply cert-manager and nginx-ingress-controller
Story: 2009101
Task: 44314
Change-Id: I99231f3f7db3d2d8eaceba137e13dea650370f71
Signed-off-by: Bin Qian <bin.qian@windriver.com>
This changes Debian package name for k8s 1.21.8 from "kubernetes" to
"kubernetes-1.21.8".
Until https://review.opendev.org/c/starlingx/integ/+/831343
is merged, version 1.21.8 is the only packaged version of
kubernetes on StarlingX Debian. In future, multiple kubernetes
versions will be supported on most, if not all, StarlingX releases.
Currently, Debian build server uses the value of 'debname' parameter in
the meta_data.yaml as the package name.
'debname' is an optional parameter in the meta_data.yaml.
If not provided, it uses package dir name as the package name
(kubernetes-1.21.8 in this case), which follows the preferred format
('kubernetes-<version>') for naming different versions of kubernetes
packages distinctly.
Test Plan:
PASS: Package builds successfully
PASS: Image builds successfully.
Story: 2009830
Task: 44638
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I46f7d9307f4254597557bb8be81ef471dcc7d73d
This introduces k8s-container-cleanup script that will be called
when containerd.service is stopped. The script detects whether systemd
state is 'stopping' due to shutdown/reboot, then stops all running
containers before the service shuts down.
During shutdown/reboot, some containers are not receiving the
SIGTERM signal. This leads to unexpected behaviour such as
generating huge coredumps.
There is an upstream issue regarding this:
https://github.com/kubernetes/kubernetes/issues/107158
The problem seems to be systemd related but this commit
addresses the problem with a workaround.
This reverts commit f3c18b0f79e3b145d378474b24d861926dd61a13.
The k8s-container-cleanup script is moved from kubelet.service
to containerd.service. The ExecStopPost that calls this script
is removed, and replaced with ExecStop in containerd.service
to call the script (in config-files repo).
The k8s-container-cleanup script requires containerd is running
in order to use crictl utility. The shutdown of kubelet and
containerd have unpredictable timing, so the cleanup must be done
in containerd.
Test Plan: On AIO-SX
PASS: Verify k8s-container-cleanup logs to daemon.log during 'stopping.
PASS: Manual change containerd/kubelet shutdown timing and verify.
k8s-container-cleanup running to completion before containerd stopped.
PASS: Reboot and verify k8s-container-cleanup running to completion.
PASS: Lock/unlock and verify k8s-container-cleanup running to completion.
PASS: Manually run spellintian tool against k8s-container-cleanup.sh.
PASS: Manually run shellcheck tool against k8s-container-cleanup.sh.
PASS: Zuul tox bashate tool against k8s-container-cleanup.sh.
Partial-Bug: 1964111
Change-Id: Ic8a9e257f861ae218a8520205eced3eaa580dd20
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
This commit fixes an issue that was seen if golang 1.17
was chosen as the toolchain to build the CNI package.
The go 1.17.5 build complains that the following vendored
modules should be explicitly required in the go.mod file:
github.com/coreos/go-iptables v0.6.0
github.com/safchain/ethtool v0.0.0-20210803160452-9aa261dae9b1
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e
If these are added to the go.mod file, a further complaint
is given that it no longer matches the information in
vendor/modules.txt
The patch files were generated by running go mod tidy for
the go.mod file, and go mod vendor for the vendor/modules.txt.
Since the bond-cni uses go 1.17 in the go directive of its
go.mod file, this commit locks down on this version to attempt
to prevent other issues from arising from new or other golang
versions.
Testing:
- CentOS build
- Debian build
- Spot check of bond-cni functionality on CentOS
Closes-Bug: 1966728
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: I14638165db48cda9b89dd666b0c8b7c0a4e8e380
This work is part of Debian integration effort.
This removes a bootstrap issue.
Docker registry service is managed by puppet.
Disable the service, otherwise it will start and listen on the same
port keystone will, thus preventing keystone from starting.
Test:
PASS: build-pkgs & build-image
PASS: bootstrap
Change-Id: Ia7a4a8525af022ebff607700c42812611f3043e8
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
To align with k8s 1.21.8 to 1.23.1,
Upgrade containerd from 1.4.6 to 1.4.11
Note:
The change pulls containerd v1.4.12 debian source package
from debian salsa. The patch 0001-revert-to-v1.4.11.patch
reverts commits between version v1.4.11 and v1.4.12.
Note that the patch has no conflicts with any of the
other patches in debian source package. So it is safe
to apply it after they get applied.
Also, it is not a strict requirement to use 1.4.11
over 1.4.12. This is just to keep in-sync with the CentOS version
of StarlingX.
Test Plan:
containerd package builds successfully
All packages build successfully
Image builds successfully
Story: 2009845
Task: 44456
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I25a15a8cac1b324411b74b9f772978270d48a664
To align with k8s 1.21.8 to 1.23.1
Upgrade runc from version 1.0.0-rc95 to 1.0.2
Dependencies are packaged in this change.
Dependencies:
golang-github-checkpoint-restore-go-criu-dev (>= 5.1.0~)
golang-github-coreos-go-systemd-dev (>= 22.3.2~)
golang-github-opencontainers-specs-dev (>= 1.0.2.66~)
golang-github-cilium-ebpf-dev (>= 0.6.2~)
Note:
As of this date, except golang-github-cilium-ebpf-dev,
all other dependencies are new and not available in bullseye
main. They are available in sid though. It was preferred to
package them from source than adding them in base-bullseye.lst.
Also, runc failed to build if newest version (0.7.0) of
golang-github-cilium-ebpf-dev is used. The exact reason is not
clear. So it was preferable to package its minimum required
version.
Test Plan:
runc package builds successfully
All packages build successfully
Image builds successfully
Story: 2009845
Task: 44456
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
Change-Id: I4139f9eb689a9e8c8e18c7a9b15fd2d592752ee5
This commit is a follow-on for d900a5b which introduced
the bond-cni plugin for Centos. In this commit we
introduce the same plugin for Debian.
Since there is no existing Debian package for the bond-cni,
we build it from source (tar.gz) to create the package.
The plugin is installed at /usr/libexec/cni/ to align
with where k8s expects plugins to be in StarlingX.
Testing:
Pass: Build
Pass: bond plugin present at /usr/libexec/cni/
Story: 2009800
Task: 44845
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: Ib9e805d587604f9b0a43a685b9b6970e5be1deb4
Testing:
* Deployed ISO with changes.
* Configured kube-cpu-mgr-policy=static.
* Verified that metrics-server were running on platform CPUs.
Partial-Bug: 1964503
Signed-off-by: Thiago Miranda <ThiagoOliveira.Miranda@windriver.com>
Change-Id: I447e4ccc113a4d0cc34a73bd71ac961305987c06
This commit uprevs the containernetworking-plugins to 1.0.1 on
Debian.
The version of this package has a Build-Depends on:
golang-github-appc-cni-dev >= 1.0.1
golang-github-vishvananda-netlink-dev >= 1.1.0.125
Currently, the Debian Bullseye release provides:
golang-github-appc-cni-dev = 0.8.1
golang-github-vishvananda-netlink-dev = 1.1.0-2
So in order to build the containernetworking-plugins 1.0.1, this
commit pulls in the appropriate versions of the dependencies
and builds them too.
The other thing to note in this commit is that the plugin install
path has been changed from /usr/lib/cni/ to /usr/libexec/cni/.
This aligns with where k8s expects to find CNI plugins on StarlingX
Testing:
- PASS: downloader -s
- PASS: full build
- PASS: install and ensure the plugins are present
at /usr/libexec/cni/
- PASS: install and ensure the package is the correct version
Story: 2009832
Task: 44635
Signed-off-by: Steven Webster <steven.webster@windriver.com>
Change-Id: Ic1c9a0d7eb2adb831c6316e1ab72d288ac805929
Simple port of two existing patches without modification to
kubernetes 1.23.1. This enables the feature to make kubelet
isolcpus allocation SMT aware.
Depends-On: https://review.opendev.org/c/starlingx/compile/+/825651
Depends-On: https://review.opendev.org/c/starlingx/integ/+/825654
Story: 2008760
Task: 44190
Test Plan:
PASS: Launch isolcpus pod and verify new kubelet logs on target host
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I2c18dea1b1f9a8a1c5e183e104a832ac872764e6
Qemu and kubernetes build failed to build under a 3 GB ramdisk.
The 6 GB ramdisk was ok.
Increase the minimum resources for these packages to 6 GB.
Closes-bug: 1964980
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I6027a4c02c15ef87a405ad300e967499c103b452
