Commit Graph

50 Commits (cc2db2aeb5f91730a6fa9c6253e683e856a4878f)

Author SHA1 Message Date
James E. Blair d6c0afb740 Revert "Test with MariaDB instead of MySQL"
This reverts commit 1c188ad440.

Reason for revert: This change increases test failures; investigation is ongoing.

Change-Id: I3a101656ee0904e1cf6c856dba845385c333260f
2023-04-13 13:38:13 +00:00
Clark Boylan 1c188ad440 Test with MariaDB instead of MySQL
I'm not sure if this is a good idea or not yet. Testing with MySQL
actually exposed the alembic transaction thing that is fixed in a
grandparent change. That said much of the world seems to be moving to
MariaDB so it might be a good idea for us to test that more explicitly?

We update bindep.txt which will install mariadb during CI job runs. We
also update our local test setup tooling to roughly match what will be
used in CI. For example the mariadb:10.6 docker image is chosen because
this is roughly equivalent to what Jammy packages.

Another good reason for this change is that Debian ARM64 does not
package MySQL and only packages MariaDB. This should give us better
compatibility for testing on architectures other than x86-64.

Change-Id: I9d00557ca5823da34278c3e9ecda2daaf13440b7
2023-04-05 08:39:34 -07:00
Clark Boylan d665053fe9 Bump up to python3.10
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.

We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.

Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.

Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.

Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
2022-07-01 15:59:48 -07:00
Albin Vass 39305393c0 Drop ambient capabilities when running bwrap
Having ambient capabilties causes bwrap to error on start [1]
unless the bwrap executable also has the setuid bit set or is run as

This can cause issues in openshift or podman unless ambient
capabilities are dropped [2].

[1] - bae85baf72/bubblewrap.c (L742)
[2] -

Change-Id: I15455fb400448d7672638f911d6cf045fa683a9b
2021-11-01 19:13:37 +01:00
Monty Taylor 3479d3eabb Update bindep for debian bullseye
The container images are based on opendevorg/python-base which is
based on python which is based on debian. Upstream python image has
updated from buster to bullseye. Once that makes it through
opendevorg/python-base, our images will start being bullseye. Make
the bindep entries forward compatible with such a future.

Change-Id: I71182e9d3e6e930977a9f983b37743ee3300ec91
2021-08-26 17:10:05 -07:00
James E. Blair 74a9c9de9b Use ZooKeeper TLS in tests
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests.  We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.

Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
2021-03-08 06:49:57 -08:00
Guillaume Chauvel d763c645d4 Fix bindep.txt: clean remaining python3.7
While upgrading tox37 to tox38, those lines were not updated/removed
they can be removed as:
-tox-py3X jobs specify python version to install during ensure-python
-other jobs are most likely using ensure-pip, which on ubuntu-bionic
install python3-venv and python3.6

Change-Id: I0abfa76d4a7057c8b9154cc111bb0aa20f4fe0bb
2020-09-29 17:16:22 +02:00
James E. Blair 041e251b7c Add the openafs-client package to the container image
This supports the afs-related roles in zuul-jobs by supplying the
unlog and vos commands.  Notably, it does not actually build and
install the afs kernel module because that is a recommended (not
required) package, and we don't automatically install recommended

Change-Id: I1a22cfa44b7eaa4dd57393e082f42843187a77d9
2020-06-29 07:16:03 -07:00
Monty Taylor eec7b303fb Add krb5-user to bindep for the images
We added openafs-krb5 but we need krb5-user for kinit.

Add DEBIAN_FRONTEND=noninteractive to the Dockerfile
to prevent the krb5-user package from asking for our
default realm.

Change-Id: Ifbef43887e541a3edc259ffaf9a75d7343c97dca
2020-06-26 08:29:47 -05:00
Monty Taylor cdc3875a05 Add openafs-krb5 to bindep
The openafs jobs in zuul-jobs depend on it being installed on
the executor.

Change-Id: Ic907fc110eb964b06a069305e3423d7c97731db4
2020-06-23 14:02:19 +00:00
Tobias Henkel b4613a0b41
Require procps also for dpkg platforms
The ps command is needed by zuul and supplied by the procps
package. This is not necessarily installed by default so require it.

Change-Id: I7d5b614ff517863d93a46789bec4065a6201d137
2020-05-12 17:15:07 +02:00
Clark Boylan 7381be2573 Update bindep to include libyaml
This updates bindep to include libyaml so that we use the C yaml parser
in testing and on our container images. This should speed up zuul in
some cases.

Change-Id: I4895f60fac248c9890ec80b03decad7973be52d5
2020-05-07 10:47:28 -07:00
Monty Taylor b52d95d1c6 Use libffi7 on ubuntu focal
Change-Id: I2741063db822e3f6d0f5c17c6b05b60e65fdfa24
2020-04-29 15:37:11 -05:00
Tobias Henkel 47671f7cc1
Install unzip on all platforms
It's already installed on rpm and apk but missing on ubuntu platforms
where it's not necessarily pre-installed so pull it in for all
platforms. This is required by fetch-sphinx-tarball of zuul-jobs so it
should be in every zuul deployment.

Change-Id: I5fb32826053ce7fe0229cea9ca89d2f7244d4044
2020-03-26 11:16:09 +01:00
Mohammed Naser 4eb1212b9d bindep: add bzip2 to all platforms
bzip2 is universal across different distributions, it's also
needed by the 'unarchive' module when running inside the executor.

Change-Id: I24ddff86b8962c902c3e541a72e8810c163cb0c9
2020-03-25 15:22:37 -04:00
Monty Taylor 957cec37f3 Remove stretch-backports from docker build
python-base is buster now, so this is not necessary. We can
just install bubblewrap and socat from buster directly.

Change-Id: If65f3c2d6367a7c79cf9d6d8f788021ba72cccd4
2020-03-12 14:17:53 -05:00
Sorin Sbarnea fa21a738be bindep: fixed wrong dep names on rpm platform
Fixed several incorrect dependency names when run on rpm platforms,
like CentOS 8 or Fedora.

Change-Id: I0d58e1abd78231bda7e1e952d71f7c5b588ad348
2020-01-20 14:30:10 +00:00
James E. Blair b2e5500cc2 Update install for buster
Add the correct libre2 package name for Debian buster, and also
update the quickstart playbook and documentation to deal with
the change in default rsa key encoding format from newer
versions of ssh-keygen.

Change-Id: I6ada88cd896d844c1171f7bcaf4691dea023d51f
2019-10-23 14:01:42 -07:00
Tristan Cacqueray 11078de772 Replace tox-py36 by tox-py37
This change enables checking Zuul with python-3.7.

Change-Id: I6b51565a81cb7d47bb971505570431ba784ec26b
2019-09-30 21:22:07 +00:00
Tristan Cacqueray 0ce741d575 bindep: add unzip and bzip2 for rpm platform
This change adds unzip and bzip2 to the rpm platform bindep so that executor
can run the fetch-sphinx-tarball role from zuul-jobs.

Change-Id: Ibe679af11b88b0912afd6d3e35a0ef9ed3fb2c8b
2019-08-25 19:53:28 +00:00
Clark Boylan 51a0635f68
Install virtualenv from source
Don't use system virtualenv. Install it from source/pypi.

Change-Id: Idea7291cb6bff9d23c87fd40b3bb45f32115b7ef
2019-03-16 11:07:58 +01:00
Tobias Henkel cd9827e664
Manage ansible installations within zuul
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
2019-03-15 09:09:16 +01:00
Monty Taylor 8a38ee711f
Replace build-essential with gcc/g++
We're not building debian packages, we're building python.

Change-Id: Ic6df129528be4f753093e1a6aeb30d7c9629d390
2019-01-22 15:57:36 +00:00
Tobias Henkel 518dbaf1f4
Add coreutils to bindep on platform apk
Alpine by default comes with busybox and its version of 'du'. This
version doesn't have the '--max-depth' switch. While this can be fixed
in zuul it further unlike the gnu version of du counts the files that
are hard linked to the cache. This makes it unsuitable for zuul and we
need to require the gnu version on alpine.

Change-Id: I39032a7ea6e7b0262eed8b1c49128e64f06029d7
2018-09-29 09:20:56 +02:00
Clint Byrum 60cabbd368 Need SSH client for Ansible
Paramiko won't cut it, as we need the ssh-agent, among other things.

Change-Id: Ic8a9544827b8aa24236caf07af7d6ac7f84808ce
2018-09-08 21:41:57 -07:00
Joshua Hesketh 793f693959 Remove redhat-rpm-config bindep
This has been fixed as an upstream dependency in the python-devel
package and can be removed from our explicit bindep list. Additionally
the package was not found or required on all rpm systems.

Change-Id: I05bd688af8353df819f27437666dbb65ddda19c2
2018-07-18 23:01:40 +10:00
Monty Taylor 95a54e3a94
Install less than alpine-sdk
alpine-sdk is a useful metapackage, but we don't actually need all of

Change-Id: I876038a40ce4a29b3e6757380b9ccbf55f199164
2018-07-16 17:31:39 -05:00
Monty Taylor c19969d033
Add alpine packages to bindep.txt
In anticipation of building containers based on python:alpine (and on
bindep understanding alpine packages) add alpine packages to bindep.txt.

We'll get testing coverage that this list works from the container build

Change-Id: I0ab864adb6bf519d7fdf0c1cf1b4a01064df5322
2018-07-12 10:11:37 -05:00
Monty Taylor 9628caffb5
Update bindep file with compile profiles
Looking ahead towards potentially using pbrx to build containers of
zuul, we want to indicate some bindep packages are needed only for
building a wheel, not for installing/running it.

Introduce a 'compile' profile that can be used to get the distro dependencies
needed for building wheels of the python dependencies. Also add runtime
library depends without a profile so that bindep -b properly reports the
packages needed for installing if wheels have been built.

Zookeeperd is moved to the 'test' profile as it is not required to be
installed locally for zuul to operate.

Change-Id: I3c8dcedddd8634661b3524966a6f228c9edde29b
2018-07-09 10:46:40 -04:00
Paul Belanger 9d6c0a9307 Add tox-py36 jobs
Now that we have ubuntu-bionic images online, we can start gating for

We only need to install bubblewrap PPA for ubuntu-xenial,
ubuntu-bionic already has bubblewrap packaged.

Update bindep.txt skipping ubuntu-xenial for bubblewrap.

Change-Id: Ic850a07001a9adef9cc459fcb107e0ebaaddf8ab
Signed-off-by: Paul Belanger <>
2018-05-07 15:07:40 +10:00
Zuul 6a1503983e Merge "Skip attempting python3-devel installation on CentOS 7" 2018-05-04 21:13:26 +00:00
Clark Boylan 7c483ff2f7 Install g++ on platform:rpm
The fb-re2 compile requires g++ which the gcc package doesn't appear to
pull in by default on Suse. During review it was mentioned that CentOS 7
doesn't pull it in either. Fix this by installing gcc-g++ on all rpm

Change-Id: Iee4da053f1d760eb3e3012cc815a2967af0e415b
2018-04-30 16:44:22 -07:00
Fatih Degirmenci 8de7313675 Skip attempting python3-devel installation on CentOS 7
python3-devel package is not available on CentOS repositories and this
causes an unnecessary error to be logged to console while installing
zuul which might confuse users.

[centos@zuul zuul]$ sudo yum install $(bindep -b) -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base:
 * epel:
 * extras:
 * ius:
 * updates:
No package python3-devel available.
Error: Nothing to do

For CentOS 7, the proposed zuul-from-scratch document instructs users to
install python35u-devel package from ius-release repo.

Change-Id: I9df218b56c59df196eff6cbd5183884c39d1ab7b
2018-04-29 23:59:48 +02:00
Tobias Henkel 0c3b8fb963 Support regex matching of github status
The github status requirements matching and trigger filter are
currently plain text matching based. This currently limits sharing of
pipeline definitions between tenants as zuul reports the status as
'<tenant>/<pipeline>'. This currently makes it necessary to define
trigger filter for each tenant [1] and completely blocks pipeline

A solution to this is regex matching which makes it possible to define
the filter once [2].

Further this enables an interesting further use case to trigger on any
successfull status [3]. This makes it easier to cooperate with other
CI systems or github apps which also set a status.

Directly use re2 as this will be used in the future for regex

[1] Trigger filter snippet
      - event: pull_request
        action: status
          - zuul:tenant1/check:success
          - zuul:tenant2/check:success
          - zuul:tenant3/check:success
          - zuul:tenant4/check:success

[2] Regex trigger filter snippet
      - event: pull_request
        action: status
          - zuul:.+/check:success

[3] Generic success filter snippet
      - event: pull_request
        action: status
          - .*:success

Change-Id: Id1b9d7334db78d0f13db33d47a80ffdb65f921df
2018-04-13 18:15:19 +02:00
Tobias Henkel 267d134b7e
Add tests for postgresql
Some deployments use postgres as database backend. In order to have
official support for that we need to add tests for it.

Change-Id: I36408c27ccb36fd0826d97f34d98b211ee075f4e
2018-04-10 15:51:25 +02:00
Paul Belanger 25d146eab8
Ensure only python3 is installed with bindep.txt
We no longer support python2.7 for zuul, we can be more specific
and now only install python3 libraries.

Change-Id: Ic31221cb55a09d9568a9f5f26c412c0559555f01
Signed-off-by: Paul Belanger <>
2018-02-14 12:36:17 -05:00
Andreas Jaeger 76605f759f Fix docs building
We need graphviz for docs building, use "doc" tag in bindep for it.
This is needed with updated jobs that look for doc tags.

Change-Id: Idc329ab113761061678c9bd31485ac5e18273e23
2017-12-18 22:21:59 +01:00
James E. Blair e86abe1ee0 Docs: add very simple architecture diagram
Change-Id: Ib0552cafd80c8a602c984cf6ca9f7f4ca8e687fe
2017-08-04 15:41:18 -07:00
James E. Blair a1733ffc11 Add redhat-rpm-config to rpm build deps
This is needed on Fedora systems to build python packages with
gcc, but for some reason is not listed as a python-devel dep.

Change-Id: Ia7479154867902b816603dc2babf6b08b91ca919
2017-06-16 15:30:58 -07:00
Paul Belanger 2989672aad Add bubblewrap to bindep /
Bubblewrap is a setuid wrapper for unprivileged chroot and namespace
manipulation. We'll be using this to run ansible-playbook from.

Change-Id: I7c6414281724d066812094d02a726df357978076
Signed-off-by: Paul Belanger <>
2017-05-16 09:01:32 -07:00
James E. Blair bf1a4f2192 Isolate encryption-related methods
Create an interface to the cryptography library so that internally
Zuul uses simple facade methods.  Unit test that interface, and
that it is compatible with OpenSSL.

Change-Id: I57da1081c8d43b0b44af5967d075908459c91687
2017-03-29 12:44:44 -07:00
Ricardo Carrillo Cruz 22994f9a09 Add per-repo public and private keys
Every project should have a public and private key to encrypt secrets.
Zuul expects them to already exist under /var/lib/zuul/keys on the
scheduler host.  If an operator manages these keys externally, they
should simply be placed there.  If they are not found, Zuul will
create them on startup and store them there so they will be found on
the next run.

The test framework uses a pre-generated keypair most of the time to
save time, however, a test is added to ensure that the auto-generate
code path is run.

Co-Authored-By: James E. Blair <>
Change-Id: Iedf7ce6ca97fab2a8b800158ed1561e45899bc51
2017-03-29 12:43:41 -07:00
Joshua Hesketh 25695cbb51 Merge branch 'master' into feature/zuulv3
Change-Id: I37a3c5d4f12917b111b7eb624f8b68689687ebc4
2017-03-06 09:40:04 -08:00
Joshua Hesketh d78b44878c Add support for sqlalchemy reporter
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.

Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <>
2017-02-06 12:47:29 -05:00
James E. Blair 498059ba28 Add Zookeeper to tests
Add a requirement on kazoo and add a Zookeeper chroot to the test

This is based on similar code in Nodepool.

Change-Id: Ic05386aac284c5542721fa3dcb1cd1c8e52d4a1f
2016-12-20 14:14:15 -08:00
Monty Taylor 1b697667d1
Stop installing mysql and postgres
Zuul doesn't have a database, so there's no need to install one, let
alone two, in its bindep.txt file. Remove them.

Change-Id: Ib1ac968d06d059d6184a8fec031cf198d0439596
2016-10-22 07:28:04 -05:00
Monty Taylor ddb1675200
Stop installing mysql and postgres
Zuul doesn't have a database, so there's no need to install one, let
alone two, in its bindep.txt file. Remove them.

Change-Id: Ib1ac968d06d059d6184a8fec031cf198d0439596
2016-10-22 07:26:59 -05:00
stephane 6444d7b977 Add pillow dependency libjpeg-dev to bindep
This should fix the error we're seeing in gate jobs:
 ValueError: jpeg is required unless explicitly disabled
 using --disable-jpeg, aborting

Change-Id: I686b21e452aad2dfb9358360137070f58dde3882
2016-10-19 13:27:25 +00:00
stephane c46caf888b Add pillow dependency libjpeg-dev to bindep
This should fix the error we're seeing in gate jobs:
 ValueError: jpeg is required unless explicitly disabled
 using --disable-jpeg, aborting

Change-Id: I686b21e452aad2dfb9358360137070f58dde3882
2016-10-18 17:58:15 -07:00
Andreas Jaeger f283e3d864 Move other-requirements.txt to bindep.txt
The default filename for documenting binary dependencies has been
changed from "other-requirements.txt" to "bindep.txt" with the release
of bindep 2.1.0. While the previous name is still supported, it will
be deprecated.

Move the file around to follow this change.

Note that this change is self-testing, the OpenStack CI infrastructure
will use a "bindep.txt" file to setup nodes for testing.

For more information about bindep, see also:

Change-Id: I7496bc019c4790d438b944631bb1632011e8f393
2016-08-12 19:31:24 +02:00