During a patch release update without a reboot, the system switches to
a new OSTree deployment. However, Calico and Multus pods that were
already running before the upgrade continue to use mount created prior
to the deployment switch. As a result, these pods remain bound to paths
from the previous OSTree deployment, even though the host has switched
to a new one. This creates an inconsistency where the host uses updated
files from the new deployment while the pods continue to use stale files
from the previous deployment.
This issue particularly affects components such as Calico and Multus,
which rely on dynamically updated data (e.g., tokens under
/etc/cni/net.d). When the token is refreshed inside the pod, the update
is not propagated to the host directory due to the stale mount.
Eventually, the outdated token expires, leading to unauthorized errors.
To address this, the solution restarts the affected components (Calico
and Multus) in scenarios where a patch update does not trigger a
reboot, ensuring that the pod mounts are reattached to the correct
OSTree deployment during post-install. Restarting the entire Kubernetes
cluster is unnecessary and time consuming, whereas restarting only the
impacted pods provides a more efficient and targeted approach.
In cases where a reboot is performed, the system naturally reconciles
itself, as mounts are recreated and aligned with the correct OSTree
deployment.
Test Plan
=========
Use the command "findmnt /etc" on the host and
"cat /proc/self/mountinfo | grep host" inside the pod to verify
which OSTree deployment is associated with the mount directory.
[PASS] On an SX system, apply a patch that does not require a reboot
and verify that after update the Calico and Multus pod mounts point to
the new OSTree deployment directory. Also verify that the pods were
restarted only once. Confirm that the Calico token refresh in
/host/etc/cni/net.d/calico-kubeconfig is reflected on the host at
/etc/cni/net.d/calico-kubeconfig, and that the Multus token refresh in
/host/etc/cni/net.d/multus.d/multus.kubeconfig is reflected on the host
at /etc/cni/net.d/multus.d/multus.kubeconfig.
[PASS] Perform a patch update using a reboot-required patch. After
deployment and system unlock, repeat the same verifications as above.
Also verify in software.log that the post-install.sh section responsible
for restarting the pods was not executed.
[PASS] Perform a patch update on a standard system using a
non-reboot-required patch. After update, repeat the same verifications
as in the first test. Also verify in software.log that the
post-install.sh section responsible for restarting the pods was
executed, and that all Calico and Multus pods were restarted once
(check with "kubectl get pods -A").
Closes-Bug: 2144903
Change-Id: Iffe18d7a26c023df02c3385578aeda9c08b90977
Signed-off-by: Ferdinando Terada <Ferdinando.GodoyTerada@windriver.com>
Ferdinando Terada
23d2727a81