Previously we pinned to 1.28/stable due to a bug that prevented
1.29/stable from working. Now we've hit a new issue with 1.28/stable on
bookworm. The fix for that appears to simply be to upgrade to
1.31/stable so we do so here. More details can be found in this GitHub
issue:
https://github.com/canonical/microk8s/issues/4361
The new version appears to return from the snap installation before the
k8s installation is fully ready to deal with add-on installation. This
occasionally produces errors like:
subprocess.CalledProcessError:
Command '('/snap/microk8s/7178/microk8s-kubectl.wrapper', 'get',
'all,ingress', '--all-namespaces')'
returned non-zero exit status 1.
Work around that with `microk8s status --wait-ready` to ensure that k8s
is up before adding addons.
While we are at it we also update the collect-kubernetes-logs role to
collect microk8s inspect output as that would've enabled us to debug the
above issue without holding nodes. We also update test jobs to trigger
when the collect-kubernetes-logs and collect-container-logs roles are
updated to ensure we get coverage from those jobs when updating these
roles.
Change-Id: I60022ec6468c2cadd723a71bbc583f20096b27dc
* This adds some extra options to the ensure-kubernetes role:
* podman + cri-o can now be used for testing
* This mode seems to be slightly more supported than the
current profiles.
* The location for minikube install can be moved.
* The use-buildset-registry role needed slight updates in order
to populate the kubernetes registry config early.
Change-Id: Ia578f1e00432eec5d81304f70db649e420786a02
* It looks like zuul-jobs-test-registry-buildset-registry-k8s-crio
is busted with Ubuntu Jammy + cri-o installed from kubic, with
errors like https://github.com/cri-o/ocicni/issues/77
(also, kubic has been wound down and cri-o has been spun off)
* cri-o in Noble uninstalls docker-ce, in a follow-up we should
clean that up and switch to a pure podman profile
* This minikube configuration is not supported, but it seems that
upstream cri-o might have made some fixes that makes it work
* Update the job to use Ubuntu Noble instead of Jammy
* Update ensure-podman for Ubuntu Noble
(podman is now part of the Ubuntu distro)
* Update the cri-o install in ensure-minikube for Ubuntu Noble and later
(cri-o is now part of k8s)
Other miscellaneous fixes and workarounds:
* k8s.gcr.io is being sunsetted, updated the test image:
https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
* Relaxed the security to run minikube from /tmp (in future,
we should set the default to /usr/local/bin)
* Updated the microk8s check-distro task for Noble
Change-Id: I3b0cbac5c72c31577797ba294de8b8c025f8c2c3
Skopeo hardcoded the docker api version for image manipulation to
version 1.22 of the api until very recently. Docker 25 sets 1.24 as a
minimum version by default introducing an incompatibility between the
tools. It isn't straightforward to install an updated skopeo everywhere
we need it (due to golang requirements). As a workaround we override
the min version to 1.22 when installing the docker daemon. This should
work until Docker 26 is released and removed the override option.
Note we also pin microk8s from latest/stable (which is currently
1.29/stable) to 1.28/stable to workaround
https://github.com/canonical/microk8s/issues/4361. This is necessary to
get the CI jobs for docker/registry/k8s testing working in order to land
this fixup.
Change-Id: I377ac84d532749eba578c4b32eb2ed6a5ce7a0c9
Add microk8s support to the ensure-kubernetes role. This installs via
a snap, and is currently only implemented for Ubuntu Jammy.
Mostly this is a straight-forward installation. I did notice though
it needs a little more time to be stable in the test, so the timeout
is bumped slightly.
microk8s is the Ubuntu "blessed" way of doing things. This should be
a better choice for Ubuntu platforms, because minikube is tightly tied
to cri-o, which is only packaged through kubic, which is currently in
some sort of deprecated but best-effort supported mode [1]. This was
inspired by an outage where the kubic gpg expired. This appears fixed
now.
[1] https://kubic.opensuse.org/blog/2022-06-10-kubic-retired/
Change-Id: Id3e31c70a35dde218e35e7c50964f8a3c0348150
This updates the ensure-kubernetes testing to check the pod is
actually running. This was hiding some issues on Jammy where the
installation succeeded but the pod was not ready.
The essence of the problem seems to be that the
containernetworking-plugins tools are coming from upstream packages on
Ubuntu Jammy. This native package places the networking tools in a
different location to those from the Opensuse kubic repo.
We need to update the cri-o path and the docker path for our jobs.
For cri-o this is just an update to the config file, which is
separated out into the crio-Ubuntu-22.04 include file.
For docker things are bit harder, because you need the cri-docker shim
now to use a docker runtime with kubernetes. Per the note inline,
this shim has some hard-coded assumptions which mean we need to
override the way it overrides (!). This works but does all feel a bit
fragile; we should probably consider our overall support for the
docker backend.
With ensure-kubernetes working now, we can revert the non-voting jobs
from the eariler change Id6ee7ed38fec254493a2abbfa076b9671c907c83.
Change-Id: I5f02f4e056a0e731d74d00ebafa96390c06175cf
Newer ansbile-lint finds "when" or "become" statements that are at the
end of blocks. Ordering these before the block seems like a very
logical thing to do, as we read from top-to-bottom so it's good to see
if the block will execute or not.
This is a no-op, and just moves the places the newer linter found.
Change-Id: If4d1dc4343ea2575c64510e1829c3fe02d6c273f
This reverts commit c4a666991207f9434a819da9bac8b3603ad3d66e.
The change was originally reverted because it broke testing on Jammy.
Jammy broke because it was attempting to install cri-o using the Bionic
method via a PPA. This is no longer valid for newer Ubuntu, and Focal
uses a different process too. We update the cri-o installation for Jammy
to match Focal and make this method the default so that new Ubuntu
releases default to modern installation methods. Bionic continues to
install from a PPA.
Additionally we bump the cri-o version from 1.15 to 1.24 to get a
version that has packages for both Focal and Jammy. This new version
requires we also install cri-o-runc separately as they don't seem to
have proper package dependencies in place between these two packages.
The crio systemd service fails to start without cri-o-runc installed.
Change-Id: Ic29576e26be094744cc1b169a3c8f0bca814f089
This was apparently not tested on Ubuntu jammy and has broken
Nodepool testing.
https://zuul.opendev.org/t/zuul/build/9ae631c37a384b4bb63515c6c5f04a00
Revert "ensure-kubernetes: pull cri-dockerd systemd from tag"
This reverts commit ad0ea28b6a2cbc8d68ee1a05e6a1d2712102dca4.
Revert "ensure-kubernetes: install cri-dockerd; fix networking"
This reverts commit 08c922fd988e98bd93d5a6d488a98e5b76797e18.
Change-Id: Ic20b8840c478b6d81626f728b8661ef5946e12d4
For ... reasons ... kubernetes deprecated docker as a container
runtime, and requires this cri-dockerd daemon to make docker cri-ish
enough to work. Install and start it so the docker path keeps
working, although long-term I guess they're saying to move away from
this (from what I read it the "none" driver will also have problems
with cgroupsv2, which makes it's future on Jaunty look interesting).
Honestly I don't really know why the cri-o now needs the
--network-plugin flag. Without it I get
X Exiting due to RUNTIME_ENABLE: unknown network plugin:
which isn't described anywhere I can see. Improvements welcome :)
Change-Id: I8ff34fa116aca14abee7e71f510bc49ffc547524
There was some recent changes to Minikube that added a hard
dependency on that file being there which are not caught by
upstream testing since they don't test with the "none" VM
driver.
This adds the file to workaround the issue so that it can
complete the bootstrap of the cluster.
Change-Id: I3869350ae620721ea4e755e50facd0291a7772d3
Minikube provides addons and we can enable specific addons by passing the --addons arg when using minikube cli
Change-Id: I8d10e3a3dde1070b5c79406285035dacb59dddc8
The minikube v1.10.x is appending the systemd-resolved conf always.
So to workaround this problem, do a patch after deployment.
Change-Id: I813151761ebeb24b3d78f2a5a6c96ca6ffd81f21
This patch refactors ensure-kubernetes to use the new generic role
to allow it to be cross-platform in the future.
Change-Id: Ic90c8973431823814c7e6b122f4ef25a67d64c2f
This adds a custom ansible-lint rule at .rules/ZuulJobsNamespaceLoopVar.py
that enforces the loop var policy described at:
https://zuul-ci.org/docs/zuul-jobs/policy.html#ansible-loops-in-roles
It also updates existing roles to follow the policy.
Change-Id: I92b2ff56a1c2702542fc07b316f1809087a4c92f
