Newer bwrap has added the ability to disable additional nested user
namespace creation from with the bwrap execution context. Take advantage
of this feature in Zuul if we are able to in order to fortify Zuul's
In particular we need two conditions to take advantage of this. 1) bwrap
must be new enough to support the feature (>=0.8.0) and 2) we must be
running with user namespaces enabled. We explicitly check for both
conditions and add the appropriate invocation flags to bwrap when the
conditions are met.
2.0 has breaking api changes:
Revert "Update git"
This reverts commit 944b9852c9.
Upstream Debian has updated git to 2.30.2-1+deb11u1 which patches git
for the issues we manually patched to cover. We don't need the manual
patch anymore and can switch to the distro hence this revert.
Reviewers should double check that the image build process installs the
expected 2.30.2-1+deb11u1 version.
Co-Authored-By: Clark Boylan <email@example.com>
skopeo has started failing with
unsupported MIME type for compression: application/vnd.in-toto+json
since the buildx v.10.0 release . The version in bullseye is a
long way behind upstream, and unfortunately there's no practical way
to backport the package (it would require also pulling in a lot of
exploded out go container dependencies).
Thus we take the alternative solution of just building it ourselves
for the executor image. I couldn't figure out how to build this with
the Debian-packaged go (I think it has something to do with the
aforementioned libraries) but it works with the upstream monolith. It
produces a binary that is linked to a few local libraries, which are
This updates git to address CVE-2022-23521.
Co-Authored-By: Jeremy Stanley <firstname.lastname@example.org>
Co-Authored-By: Clark Boylan <email@example.com>
This updates the openshift client install to use the latest stable
release. Hashes of the oc and kubectl command remain the same which
should continue to allow us to avoid copying both files.
Note we don't fetch the client from the stable-4.11/ path because the
versions of the client under this path are updated when the stable
version updates. Instad we fetch it from the permanent location for the
current stable release (4.11.20/).
These binaries are about 115MB each and we copy both of them.
Fortunately they are identical according to hashing routines so we can
save space by copying them once and using a symlink. We choose to make
`oc` the canonical file as these binaries come from openshift.
This adds python 3.11 testing and drops python3.10 in order to keep
testing only the bounds of what Zuul supports. Note that currently the
python 3.11 available for jammy is based on an RC release. This should
be fine as we do functional testing with a released python 3.11 and that
is what people will consume via the docker images.
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.
We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.
Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.
Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.
We had been using version 14 which is the previous LTS. Now there are
npx browserslist@latest --update-db
running out of memory. Update to the current nodejs LTS version to
ensure we are running an up to date runtime that hopefully performs more
consistently with the browserslist command.
Debian bullseye includes skopeo, so we can drop the kubic repository
(which as I write this is having synchronization errors) in favor of
just using the version from the underlying OS.
Bullseye is out and Buster is old news :) Bump up the base image that we
build Zuul on to Bullseye from Buster. The python version remains the
same but this gives us a more up to date userland. In particular git
goes from 2.20 to 2.30.
We can use more than one builder image. Use the node image for
node in the python builder image.
Git versions between 2.18 and 2.26 have version 2 protocol support
but it's not enabled by default. Starting in 2.26, it is the
default. V2 is more efficient at negotiating refs and can reduce
the time spent updating a repo by 50% on large repos.
This adds a setting to the container images to specify that v2
should be used. The images are currently built with git 2.20, which
is in the range where this is needed.
After dropping support of Ansible 2.7 which has compatibility issues
with python 3.8 we now can finally upgrade to Python 3.8 which has
improvements regarding performance and memory usage.
There was a bug in opendevorg/python-builder that prevented extras from
being installed properly. Zuul worked around this with an explicit step
to install those extras. The depends on of this change fixes the
python-builder bug so we can remove the workaround from Zuul's
The js content tarball creation was broken. Instead of leaving the
symlink to a non-existing directory which gets created during python
setup we should just remove that symlink as well and create the
symlink and the static dir during the python setup. This way nothing
This reverts commit eb7b18b38e.
Fixed error observed locally when trying to reproduce CI command failure:
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
AFS filespace is organized into cells or administrative domains. Each
workstation belongs to one cell. Usually the cell is the DNS domain name of
AFS cell this workstation belongs to: ^C
We added openafs-krb5 but we need krb5-user for kinit.
Add DEBIAN_FRONTEND=noninteractive to the Dockerfile
to prevent the krb5-user package from asking for our
virtualenv 20.0.24 creates ~/.local/share/virtualenv with the
seed packages needed for making virtualenvs per-python version.
Creating empty virtualenvs is quick, so run those in sequence
to avoid race possibilities. Then, we can still run the
installs into the virtualenvs in parallel.
We also fix a bug in the console stream functional jobs and install pip
with the use of ensure-pip. This is necessary because the virtualenv
fix runs the stream functional jobs and the update to the stream
functional jobs relies on working docker images.
We are consuming OpenDev's default python-builder and python-base images
which happen to be python3.7 today. Zuul specifically wants python3.7 so
we should explicitly use OpenDev's 3.7 tags of these images. This way if
OpenDev updates the default to 3.8 zuul can continue to assert its
dependency on 3.7.
We suspect a memory leak in python 3.7. BMW saw one and reverted
to 3.6 and has been better. OpenDev is seeing one which seems to
be improved by sending SIGUSR2 signals to the process, which makes
us think there may be an issue in the GC.
Try updating to 3.8. If it fixes it, we can just shrug and say
"3.7 lol". If it doesn't, we can follow BMW's lead and try 3.6.
When we get a pod from nodepool, this starts a kubectl port-forward
on the pod so that zuul-console and the normal method of streaming
command output will work.
The current options stated as 'do not run as a daemon' is actually
'run in debug mode in foreground'. When running in container we
actually want an option for running normally in foreground. Thus add a
new option -f for foreground operations and change the docker imaged
to use this accordingly.
context, so just run yarn explicitly.
This installs oc and kubectl (really oc masquerading as kubectl)
into the zuul-executor container image, so that Ansible kubectl
connections work as expected.
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.
The actual usage of multiple ansible versions will be done in
For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.
Most configuration options depend on storing state data inside
the /var/lib/zuul path which does not exist by default, causing
a lot of configurations to not work out of the box.
This patch creates the folder in the zuul base image, which will
reduce the number of options to be moved around to store state.
The Service Workers seem to be consistently causing issues for people
that are strange, meaning many of our deployers are disabling them.
Since they aren't super necessary for the Zuul use case, change the
default behavior to be to disable them instead of enable them.
So that people can re-use the Dockerfiles to build zuul images
but with different flags set, plumb the env vars through here
as ARG entries.
Also, fix 2 doc references that were misspelled.