This adds new style mirror_info handling to use-docker-mirror to give us
greater control over whether or not docker hub should be mirrored. We
ignore old style configuration if new style is present which gives us
this control. Otherwise we fallback to the old behavior.
We also update the ensure-docker test jobs to be triggered by updates to
the use-docker-mirror roles as ensure-docker includes this role. We
should get decent functional testing coverage this way.
Change-Id: Ia1b216a6dd68bcafbe599777037c5d7b1b3e8201
the openvswitch.openvswitch collection is removed from Ansible packages
starting with Ansible 11. This causes ansible-lint to correctly not find
the openvswitch_bridge module when ansible-lint runs with Ansible 11.
Workaround this by capping Ansible used by ansible-lint to <10 and leave
a note about the module going away where we use it.
Change-Id: Id2d4e4f59c7d7e595c5458bc8717146c2326c573
The trademark for the logo, as filed, is specified as dark blue.
That logo needs to be displayed in a user guide or manual or
alongside a direct download of the software in order to demonstrate
continued use of the mark, and the foundation's trademark lawyers
have expressed concern that a white version is insufficient (but
they seem to be okay with the current background color as long as
the lines of the logo are colored similar to the version on file).
This is a copy of the equivalent change from the zuul repository.
Co-Authored-By: James E. Blair <jim@acmegating.com>
Depends-On: https://review.opendev.org/934443
Change-Id: I2de26048c3a4cb8cb6b67bf786b526963c6ec04b
For some reason (unknown really for us) triggering webhook with http
basic auth using Ansible's uri module started recently failing when it
is run on some operating systems, like e.g. Ubuntu Noble.
Let's switch to use curl command directly to trigger that webhook
instead.
Change-Id: Idbf643ea27220504ac9e37eaf9f18930d2fc08ab
If you need to run native arm64 builds, you can take advantage
of this change which will rely on the remote builders in order
to build things natively giving a significant speed up in
container build time.
Change-Id: I962bb2357a2c458d5e72b334b4fe36b55b034864
The blockdiag/seqdiag set of tools and their sphinx extensions are no
longer maintained. This hasn't been a huge issue until we started
running jobs on Python3.12 as we need to run an older version of Pillow
to support these tools and that needs special libs to build wheels on
python3.12.
Rather than continue to try and make old unmaintained tools work we
switch to graphviz which is maintained and has support built into
sphinx. This does require us to install graphviz as a system dep but
that seems like a reasonable tradeoff for using supported tooling.
The resulting graph specifications are also slightly more verbose.
Co-Authored-By: James E. Blair <jim@acmegating.com>
Change-Id: I2d1e4c3d648723402aae2d87fb3233f4418d5003
This counts the open file handles and inodes. This may be useful
(after establishing a baseline) for evaluating ulimit errors.
Change-Id: I6d5c67d7c5c03d4aa7cd88b2238163cc729d9782
We removed the default value, because having a default value actually
makes no sense at all. To be helpful for any transitions, add a runtime
check that the variable is set.
Also, while we're at it, update the docs to indicate that the parameter
is required.
Change-Id: I1e18ea51d9d56561608ff241d71b63965c4f78bd
This job that tests the zuul-jobs tox role runs various tox targets
including the docs target. This means we need to install dependencies
for doc building. On Ubuntu Noble this include libjpeg-dev for Pillow
wheel building and we fail without this profile installed.
We keep the default profiles of compile and test because we also run
other targets that likely depend on these profiles.
Change-Id: Ifa3495488f35b1fbe4fc665c4d0ac5ed8adb33aa
The ensure-nodejs role defaults to install nodejs 6 which produces this
error currently:
Failed to update apt cache: W:The repository
'https://deb.nodesource.com/node_6.x noble Release' does not have a
Release file., W:Data from such a repository can't be authenticated
and is therefore potentially dangerous to use.
We need to make a few changes to bring this ensure-nodejs role up to
modern expectations for nodesource usage. First we drop the default
nodejs version from ensure-nodejs. Everyone is already setting this
value to make this role work or they are broken and will need to change
something anyway. This gets us off of the nodejs update treadmill in
this role.
Then with nodejs 16 and newer there is a new gpg key and no deb-src
packages so we need to change the apt configuration if using 16 and
newer. We make these changes to match the corresponding setup_16.x etc
scripts from nodesource.
Change-Id: I0d5c93e4fbcee0be2cc477bf9f625e419a2b9bd1
We don't need to name every play; in the Zuul context, they are
usually pretty self-evident. It might be nice, but it doesn't
seem necessary to require it.
We really don't need to care about using upper or lower case.
The key-order seems very arbitrary and counterproductive to make
all our developers memorize someone else's arbitrary preference
for ordering.
Change-Id: I49455b6946d5d9b6bffd58420fea586ecc6c5f80
Previously we pinned to 1.28/stable due to a bug that prevented
1.29/stable from working. Now we've hit a new issue with 1.28/stable on
bookworm. The fix for that appears to simply be to upgrade to
1.31/stable so we do so here. More details can be found in this GitHub
issue:
https://github.com/canonical/microk8s/issues/4361
The new version appears to return from the snap installation before the
k8s installation is fully ready to deal with add-on installation. This
occasionally produces errors like:
subprocess.CalledProcessError:
Command '('/snap/microk8s/7178/microk8s-kubectl.wrapper', 'get',
'all,ingress', '--all-namespaces')'
returned non-zero exit status 1.
Work around that with `microk8s status --wait-ready` to ensure that k8s
is up before adding addons.
While we are at it we also update the collect-kubernetes-logs role to
collect microk8s inspect output as that would've enabled us to debug the
above issue without holding nodes. We also update test jobs to trigger
when the collect-kubernetes-logs and collect-container-logs roles are
updated to ensure we get coverage from those jobs when updating these
roles.
Change-Id: I60022ec6468c2cadd723a71bbc583f20096b27dc
It's highly likely that folks may want to use YAML anchors to
build up list of DIB elements. To aid in that, allow the value
to be a list of lists and automatically flatton it.
Change-Id: I55b9cb16951b51da32f99ca5858b75217951b279
It would be useful especially when ec2 fleet api is configured,
and the instance type is unknown in advance.
Change-Id: Ibcdade5cfffd13fddd95e797c60c5327bb34fdb6
Fstrings are not supported in python3.5 which is in use on Xenial.
We don't claim to support Xenial, but this is an easy regression
to avoid.
Also, add test jobs for this role so that we get feedback before
copying it to the prod roles.
Also, add a xenial test job to exercise it since we still have
Xenial nodes available.
Change-Id: Ifc773aa688adb1a01cfe691b3bdca0b3086658cd
This adds a role convert-diskimage which uses the qemu-img tool to
convert diskimages from one format to another. Currently supported image
formats are raw and qcow2.
Change-Id: I4770af04c37f39e0cce23d5dd59ead744bed7d74
This adds a role variable to configure the diskimage-builder environment.
This allows users a choice of using the Ansible "environment" argument,
or using a variable. The variable may be particularly useful since it
allows full configuration of the role from a Zuul job definition.
Change-Id: I68542f13454b4f2e2e9bb8d356feefddba23d8f2
* This adds some extra options to the ensure-kubernetes role:
* podman + cri-o can now be used for testing
* This mode seems to be slightly more supported than the
current profiles.
* The location for minikube install can be moved.
* The use-buildset-registry role needed slight updates in order
to populate the kubernetes registry config early.
Change-Id: Ia578f1e00432eec5d81304f70db649e420786a02
* It looks like zuul-jobs-test-registry-buildset-registry-k8s-crio
is busted with Ubuntu Jammy + cri-o installed from kubic, with
errors like https://github.com/cri-o/ocicni/issues/77
(also, kubic has been wound down and cri-o has been spun off)
* cri-o in Noble uninstalls docker-ce, in a follow-up we should
clean that up and switch to a pure podman profile
* This minikube configuration is not supported, but it seems that
upstream cri-o might have made some fixes that makes it work
* Update the job to use Ubuntu Noble instead of Jammy
* Update ensure-podman for Ubuntu Noble
(podman is now part of the Ubuntu distro)
* Update the cri-o install in ensure-minikube for Ubuntu Noble and later
(cri-o is now part of k8s)
Other miscellaneous fixes and workarounds:
* k8s.gcr.io is being sunsetted, updated the test image:
https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
* Relaxed the security to run minikube from /tmp (in future,
we should set the default to /usr/local/bin)
* Updated the microk8s check-distro task for Noble
Change-Id: I3b0cbac5c72c31577797ba294de8b8c025f8c2c3
This updates test-prepare-workspace-git to use a new Ansible module
for its work. This module supports each of the three main tasks
with a single Python invocation for all projects, rather than
using Ansible to loop over them.
Within the python invocation, a ThreadPoolExecutor with 10 workers
is used to execute the setup processes with increased parallelism.
This should greatly improve the workspace sync speed for jobs with
large numbers of repos.
A simple local test with the 16 Zuul project repos reduces the
workspace-prep time to less than 50% of the current time.
This removes the mirror_workspace_quiet role variable, since detailed
information about the repo state can now be found in the repo state
JSON file.
Otherwise, the actual git commands used to prepare the repos should
be identical.
Change-Id: Ib4b6bb379fe656ac9109b8e6073eff8c28c7eaee
The get_md5 parameter was removed with ansible 9.
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_9.html#id44
If it is being used the following error appears:
"Unsupported parameters for (stat) module: get_md5..."
Unrelated, but also blocking testing/merging of this change, the
Ansible version specs for older python versions is loosened
to allow installing older versions of Ansible on test nodes (like
focal) that have older pythons that are unsupported by newer Ansible.
Change-Id: I99dd4f16fde659d84eb3dfa191557b3d9508b0fb
The default wheel_mirror(including major.minor) no longer works for Debian,
Fix it by using just major version like debian-11-x86_64. Similar was
fixed in [1] but missed fixing configure-mirrors.
[1] https://review.opendev.org/c/openstack/project-config/+/897545
Change-Id: I4194f18a06527d8af8922f3baf8766a7148e23fa
This runs the same commands on the git repo regardless of whether
it was cloned or if the role found it there already. Since the
purpose of the role is to mirror the workspace repos from the
executor to the remote node, this will produce more consistent
behavior.
Note that anyone somehow relying on, say, the origin being set
outside of this role may encounter a behavior change. It is expected
that anyone manipulating a repo that is also managed by this role
would perform those manipulations idempotently after running this
role.
Change-Id: I428bf2980a526919d5b154c585943be92d4c1cfa
Commit 8003cdc76ca177061b1a462d07efaff83e18491b causes problems
if the remote repo already exists (e.g., the worker node is static
and not ephemeral) because it unconditionally removes settings
which are only conditionally set if the workspace is newly cloned.
Fix that by remove the Ansible "creates" check from the task and
executing the set calls unconditionally (but also, recreate the
functionality of the create check for the cloning part of the
task, which is what we're really trying to avoid).
This will run a few extra command such as clearing the bare flag
and also resetting the origin remote. That should be fine in
this role since we expect it to do whatever it takes to make the
remote repo the same as the local one.
Also, resync test-prepare-workspace-git.
Change-Id: Ife12992df9ce2b0ce199b3980a4baa255cb0f28a
