We don't need to name every play; in the Zuul context, they are
usually pretty self-evident. It might be nice, but it doesn't
seem necessary to require it.
We really don't need to care about using upper or lower case.
The key-order seems very arbitrary and counterproductive to make
all our developers memorize someone else's arbitrary preference
for ordering.
Change-Id: I49455b6946d5d9b6bffd58420fea586ecc6c5f80
It's highly likely that folks may want to use YAML anchors to
build up list of DIB elements. To aid in that, allow the value
to be a list of lists and automatically flatton it.
Change-Id: I55b9cb16951b51da32f99ca5858b75217951b279
Fstrings are not supported in python3.5 which is in use on Xenial.
We don't claim to support Xenial, but this is an easy regression
to avoid.
Also, add test jobs for this role so that we get feedback before
copying it to the prod roles.
Also, add a xenial test job to exercise it since we still have
Xenial nodes available.
Change-Id: Ifc773aa688adb1a01cfe691b3bdca0b3086658cd
This adds a role convert-diskimage which uses the qemu-img tool to
convert diskimages from one format to another. Currently supported image
formats are raw and qcow2.
Change-Id: I4770af04c37f39e0cce23d5dd59ead744bed7d74
This adds a role variable to configure the diskimage-builder environment.
This allows users a choice of using the Ansible "environment" argument,
or using a variable. The variable may be particularly useful since it
allows full configuration of the role from a Zuul job definition.
Change-Id: I68542f13454b4f2e2e9bb8d356feefddba23d8f2
* This adds some extra options to the ensure-kubernetes role:
* podman + cri-o can now be used for testing
* This mode seems to be slightly more supported than the
current profiles.
* The location for minikube install can be moved.
* The use-buildset-registry role needed slight updates in order
to populate the kubernetes registry config early.
Change-Id: Ia578f1e00432eec5d81304f70db649e420786a02
* It looks like zuul-jobs-test-registry-buildset-registry-k8s-crio
is busted with Ubuntu Jammy + cri-o installed from kubic, with
errors like https://github.com/cri-o/ocicni/issues/77
(also, kubic has been wound down and cri-o has been spun off)
* cri-o in Noble uninstalls docker-ce, in a follow-up we should
clean that up and switch to a pure podman profile
* This minikube configuration is not supported, but it seems that
upstream cri-o might have made some fixes that makes it work
* Update the job to use Ubuntu Noble instead of Jammy
* Update ensure-podman for Ubuntu Noble
(podman is now part of the Ubuntu distro)
* Update the cri-o install in ensure-minikube for Ubuntu Noble and later
(cri-o is now part of k8s)
Other miscellaneous fixes and workarounds:
* k8s.gcr.io is being sunsetted, updated the test image:
https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
* Relaxed the security to run minikube from /tmp (in future,
we should set the default to /usr/local/bin)
* Updated the microk8s check-distro task for Noble
Change-Id: I3b0cbac5c72c31577797ba294de8b8c025f8c2c3
This updates test-prepare-workspace-git to use a new Ansible module
for its work. This module supports each of the three main tasks
with a single Python invocation for all projects, rather than
using Ansible to loop over them.
Within the python invocation, a ThreadPoolExecutor with 10 workers
is used to execute the setup processes with increased parallelism.
This should greatly improve the workspace sync speed for jobs with
large numbers of repos.
A simple local test with the 16 Zuul project repos reduces the
workspace-prep time to less than 50% of the current time.
This removes the mirror_workspace_quiet role variable, since detailed
information about the repo state can now be found in the repo state
JSON file.
Otherwise, the actual git commands used to prepare the repos should
be identical.
Change-Id: Ib4b6bb379fe656ac9109b8e6073eff8c28c7eaee
The get_md5 parameter was removed with ansible 9.
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_9.html#id44
If it is being used the following error appears:
"Unsupported parameters for (stat) module: get_md5..."
Unrelated, but also blocking testing/merging of this change, the
Ansible version specs for older python versions is loosened
to allow installing older versions of Ansible on test nodes (like
focal) that have older pythons that are unsupported by newer Ansible.
Change-Id: I99dd4f16fde659d84eb3dfa191557b3d9508b0fb
The default wheel_mirror(including major.minor) no longer works for Debian,
Fix it by using just major version like debian-11-x86_64. Similar was
fixed in [1] but missed fixing configure-mirrors.
[1] https://review.opendev.org/c/openstack/project-config/+/897545
Change-Id: I4194f18a06527d8af8922f3baf8766a7148e23fa
This runs the same commands on the git repo regardless of whether
it was cloned or if the role found it there already. Since the
purpose of the role is to mirror the workspace repos from the
executor to the remote node, this will produce more consistent
behavior.
Note that anyone somehow relying on, say, the origin being set
outside of this role may encounter a behavior change. It is expected
that anyone manipulating a repo that is also managed by this role
would perform those manipulations idempotently after running this
role.
Change-Id: I428bf2980a526919d5b154c585943be92d4c1cfa
Commit 8003cdc76ca177061b1a462d07efaff83e18491b causes problems
if the remote repo already exists (e.g., the worker node is static
and not ephemeral) because it unconditionally removes settings
which are only conditionally set if the workspace is newly cloned.
Fix that by remove the Ansible "creates" check from the task and
executing the set calls unconditionally (but also, recreate the
functionality of the create check for the cloning part of the
task, which is what we're really trying to avoid).
This will run a few extra command such as clearing the bare flag
and also resetting the origin remote. That should be fine in
this role since we expect it to do whatever it takes to make the
remote repo the same as the local one.
Also, resync test-prepare-workspace-git.
Change-Id: Ife12992df9ce2b0ce199b3980a4baa255cb0f28a
In cases where the HEAD branch of a repository is deleted,
prepare-workspace-git will be unable to propagate that deletion from
the executor repos to a clone of a cached repo on a remote node.
Correct this by unsetting the receive.delyDeleteCurrent flag. This was
correctly set in mirror-workspace-git, but did not make it into
prepare-workspace-git.
Change-Id: I159fb1f9f9cee873466f11be2f155bed6892472e
Poetry (https://python-poetry.org) is not declared as a dependency for a
Python project, it must be available somehow in the system. This role
installs it if missing.
- Latest version is installed, unless `ensure_poetry_version` is
informed.
- The installed executable path is set as the `poetry_executable` fact.
- The `/usr/local/bin/poetry` symlink can also be created if
`ensure_poetry_global_symlink: true`.
This new role is basically a copy of the `ensure-nox` role with the
symlink creation snippet taken from the `ensure-tox` role.
The commit adding `ensure-nox` (77b1b24) has been taken as an example of
the necessary changes when adding a new role.
Change-Id: I5592d38d415a9d74055348406653b69f110541ae
CentOS 8 Stream is EOL and these jobs are no longer functional. We would
like to clean up the test nodes entirely as a result of this. Part of
this cleanup is either deleting existing jobs that rely on CentOS 8
Stream or converting them to run on a different platform. In this case
we update the fips role job to run on CentOS 9 Stream.
Change-Id: Ifb7d6deaf529e21c257590d63938026584bee7ae
We use mimetypes to set file mime types for upload to log server
locations. We also override yaml files mime type to text/plain when
doing so. If we then call mimetypes.init() again this overrides the
previous yaml update to text/plain. This wasn't a problem until
python3.12 but python3.12 (on Ubuntu Noble specifically) seems to import
test cases in a different order which results in the mimetypes.init()
call in generate_manifest.py overriding the yaml mimetype set by the log
upload test cases.
Simply check if mimetypes is already inited before we init it again
which should avoid the problem entirely.
Note that this is likely to only ever by a testing issues as typically
ansible wouldn't import all of this code together but the unittest
runner does.
Change-Id: Ifb9137ddd89713cad546129c462ad94315100940
We have support for installing python from pyenv, but it currently
requires setting the python version explicitly as an argument. If
the repo in question has a .python-version file, we shouldn't need to
require the user to provide that version a second time. Instead, we can
read from the file for the install step.
Change-Id: Ic4c2d3fc7f55169cec5211010fc3a9622fa324d1
Python 3.12 is a thing now and is present in a stable Linux distro
release (Ubuntu Noble). It seems reasonable for people to want to run
nox and tox against py312 targets. Let's make it easy for them and add
jobs to do that for them.
Change-Id: I9d644cfbe65b92207a5fe2ad6dd950093bda87dd
We were pinning ansible to 2.8 for unittesting but Zuul currently only
supports Ansible 8 and 9. Pin Ansible to 8 in order to better test what
we expect people are using in the wild. This will also enable the
testing of Zuul and Ansible and Zuul Jobs with newer python versions
like 3.12.
In order to run testing with Ansible 8 intsead of 2.8 we drop testing
against python2.7 and python3.8 as neither of them can install an
Ansible version this new.
Change-Id: Icd563def65dcfd40b174218cc4e2b94e0230c374
The package repos for centos-8-stream are empty now, so it can no longer
be deployed. Ubuntu Bionic is EOL and about to be deleted from the
opendev infrastructure, too.
Change-Id: I5e14957f2c866fc3e337ac4051692c68b450ceba
OpenDev has recently added ubuntu-noble nodes, so let's test our
standard library on that platform now.
Change-Id: I8d49696812e6b0c67723fba9242ea6a4f5cd5026
There are a number of updates we make for Ubuntu Noble in this commit.
1. Remove python2-dev from bindep for Noble as Noble appears to have no
python2 runtime options.
2. Add libjpeg-dev to bindep for noble because Pillow doesn't build
python3.12 wheels for Pillow<10 which we currently depend on. This
means we need to build from source and that depends on libjpeg-dev.
3. We remove double bracket wrappers from ansible vars in ansible
assertion blocks. Having them results in errors like:
Conditional is marked as unsafe, and cannot be evaluated.
4. We update rust testing to explicitly install pkg-config before
building python cryptography. This tool is required to build
cryptography from source and is no longer being pulled in either
by the base images or build-essential meta pacakge.
5. Add an Ubuntu-24.04 tasks file for the ensure-skopeo roles so that
we try to install skopeo using distro packages or build from source
and don't use Kubic which only has packages for old Ubuntu releases.
Change-Id: I388710ce40dc757ada4de819a9c3c59fc32fb07a
We need this specific fix [1] to target error:
from collections import Sequence, Mapping
ImportError: cannot import name 'Sequence' from 'collections'
[1] https://github.com/23andMe/Yamale/commit/
0ddd618c8ba7aa6df146a840b15a1285c476e10e
Change-Id: I04f3221624bdf88803b9e5f4a9fa3eabade756b1
As a followup to I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
We switch buildset-registry jobs to debian bookworm which has new enough
golang to build the latest skopeo version. Latest skopeo is used in
order to get api version negotiation behavior which is necessary for
talking to modern docker (version 25 or newer).
Change-Id: Ie673ef6724b0a40e3cfb2ba83e90d566e1f1837c
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
This reverts *parts* of commit 9519fafd102017e54f4d543f25ca4398f1ec67b8.
Specifically, this reverts the main fix regarding DOCKER_MIN_API_VERSION.
Reason for revert: Docker 26.0.0 is out, which drops the old API
This patch additionally modifies the Zuul config in two ways:
* disables buildset-registry jobs because they are children of
opendev-buildset-registry (a base-job in opendev) and thus
do not pick up new changes - they are reenabled in a followup
* disables crio jobs because they have their repo seemingly broken
Change-Id: I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
Co-Authored-By: Radosław Piliszek <radek@piliszek.it>
In case of aws spot instances, the node can drop the connection anytime.
Ansible loops do not fail when the first item fails, instead they execute all items before reporting the failure.
This leads to a big overhead, if we sync repositories, when the node has dropped the connection beforehand.
Therefore i used include_tasks, which fails for the first unsuccessful item.
Change-Id: Id6079a2cda10a825384d52d47750d9c05d323e00
While requests-oauthlib 1.4.0 reports compatibility with python2.7
evidence points to the contrary. Specifically:
import requests_oauthlib as oauth
File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/__init__.py", line 5, in <module>
from .oauth1_session import OAuth1Session
File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/oauth1_session.py", line 1, in <module>
from urllib.parse import urlparse
ImportError: No module named parse
The azure-storage-blob library depends on this oauthlib which is needed
for log upload role testing. We address that by pinning the lib under
python2.7.
Change-Id: Ia27f2df166ccb33dcc93334d884411670ac859f7
CentOS 7 is going EOL in the near future. Due to a lack of demand/need
within OpenDev and the upcoming EOL OpenDev has decided to drop CentOS 7
test nodes entirely. This will free up room for OpenDev to build and
mirror distro releases that are more relevant today. In preparation for
this removal we drop the zuul-jobs CentOS 7 testing.
Change-Id: Id49d28dd426ecbb3470e234d3b67875a6b097112
