Commit Graph

49 Commits (master)

Author SHA1 Message Date
James E. Blair 86e86ffbd8 Add a release note about Python 3.11
This is now the only version of Python tested with Zuul.  Note that
change and also update the pypi classifiers.

That requires that we run all the nox jobs with Python 3.11, so they
are updated as well.

Change-Id: I9bb7514d5eab32fed814a5a054cc6be642e91ab4
2023-07-27 17:09:54 -07:00
James E. Blair 6c8855ede4 Add noxfile and switch to nox
Tox v4 behaves significantly differently than v3, and some of the
more complex things we do with tox would need an overhaul to
continue to use it.  Meanwhile, nox is much simpler and more
flexible, so let's try using it.

This adds a noxfile which should be equivalent to our tox.ini file.
We still need to update the docs build (which involves changes to
base jobs) before we can completely remove tox.

Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
2022-12-20 08:57:53 -08:00
Clark Boylan 9b2b8f82ff Add Python 3.11 testing
This adds python 3.11 testing and drops python3.10 in order to keep
testing only the bounds of what Zuul supports. Note that currently the
python 3.11 available for jammy is based on an RC release. This should
be fine as we do functional testing with a released python 3.11 and that
is what people will consume via the docker images.

Change-Id: Ic5ecf2e23b250d3dbf592983b17ec715d6e9722e
2022-10-27 16:47:14 -07:00
Stephen Finucane 694fb6df1d setup: Replace dash-separated config
Resolve the following warnings:

  UserWarning: Usage of dash-separated 'description-file' will not be
  supported in future versions. Please use the underscore name
  'description_file' instead

Change-Id: Ibd13b66a1283714c2ceaecdfc0e1de7a5338df5f
Signed-off-by: Stephen Finucane <>
2022-08-19 11:25:30 +00:00
Zuul 78b14ec3c1 Merge "Bump up to python3.10" 2022-07-05 10:05:37 +00:00
Clark Boylan d665053fe9 Bump up to python3.10
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.

We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.

Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.

Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.

Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
2022-07-01 15:59:48 -07:00
Matthieu Huin 57c78c08e1 Clarify zuul admin CLI scope
We have two CLIs: zuul-client for REST-related operations, which cover
tenant-scoped, workflow modifying actions such as enqueue, dequeue and
promote; and zuul which supercedes zuul-client and covers also true admin
operations like ZooKeeper maintenance, config checking and issueing auth tokens.
This is a bit confusing for users and operators, and can induce code

* Rename zuul CLI into zuul-admin. zuul is still a valid endpoint
  and will be removed after next release.
* Print a deprecation warning when invoking the admin CLI as zuul
  instead of zuul-admin, and when running autohold-*, enqueue-*,
  dequeue and promote subcommands. These subcommands will need to be
  run with zuul-client after next release.
* Clarify the scopes and deprecations in the documentation.

Change-Id: I90cf6f2be4e4c8180ad0f5e2696b7eaa7380b411
2022-05-19 15:35:30 +02:00
Clark Boylan 2cc01ffc5a Set python-requires to >=3.8
We actually reqire 3.8 or newer now so ensure that is reflected in the
package metadata.

Change-Id: Ia6ef56729b4ca3e93a694bdf6ba375ae4c34e86a
2022-05-06 13:49:45 -07:00
James E. Blair 6360bf85a8 Update testing to Python 3.8 and 3.9
Python 3.6 is EOL and Ansible 5 does not support it.  Bump our
testing to 3.8 + 3.9 to match Nodepool.

Change-Id: Ica6094c05afb87cd5bbc8752c6362d89c13eca0a
2022-04-14 13:33:39 -07:00
Zuul 309757e0f6 Merge "Include database requirements by default" 2021-03-09 23:24:34 +00:00
Paul Belanger c1e39ff457 Include database requirements by default
An SQL database is an additional requirement so we should also include
the client libraries for it.

Change-Id: Id3a8b55928d9ce9e8764f18a64a42bb2f14f5ecb
Signed-off-by: Paul Belanger <>
2021-02-23 16:38:47 -05:00
James E. Blair 1ea4ffb9ea Add python-logstash-async to container images
This adds the python-logstash-async module to the list of packages
that should be present in container images.  This allows operators
to configure direct-to-logstash logging from Zuul.

Change-Id: I6764876c7e64afa94dcdf8f49215cf7abc607a57
2021-02-18 14:48:55 -08:00
James E. Blair f1e7ffab93 Remove zuul-migrate
This was intended as a one-time helper program to help people upgrade
from Zuul v2 to v3.  It did not cover all use cases, and has not been
kept up to date or improved.  It's time to remove it before the v4

Change-Id: I12cdcedb5baabd8fa0937a6ea21590259093ead1
2021-02-17 09:22:46 -08:00
Sorin Sbarnea c911e26fa6 Decouple mypy from tox
This allows mypy to produce the same results regardless how is called.
For example IDEs will produce the same results as tox -e linters
instead of displaying violations which the core team decided to

Change-Id: I20eeafc0549f78e3d4f9636793cfe031389b6bbe
2020-12-09 17:04:45 +00:00
Tobias Henkel 6ab497ad3d
Bump mypy to 0.790
The currently used version 0.740 throws syntax errors when used with
python 3.9.

Change-Id: If5cd0da2d5a77126e9ac4c4d6db7a6552f1bd067
2020-12-09 16:28:27 +01:00
Tobias Henkel d676b6d6d9
Drop support for python 3.5
There are now increasing numbers of dependencies like cryptocraphy
that will drop support for Python 3.5 soon. Thus we should consider
dropping the support for Python 3.5 as well.

Change-Id: I830ec3e97cfaac5d336d02755fe788572f36fb07
2020-09-04 16:21:10 +02:00
Monty Taylor 58902225dd Add yappi and objgraph to container image
If people want to do debugging of scheduler issues, these
are useful to have installed.

Change-Id: I93e949f75d38cc86c799dc52a1b939e61e81d74b
2020-04-05 08:41:01 -05:00
Jeremy Stanley 7e53f87d06 Declare support for Python3.8
Replace our tox-py37 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.

Change-Id: I4b382bb89847d3ae2b20292d923c3c2211dfb254
2020-03-11 20:22:21 +00:00
Tobias Henkel 8e3d60567b
Cleanup executor specific requirements
Ara and openstacksdk are installed into the ansible virtualenvs now so
there is no need to pull them as executor specific dependencies.

Change-Id: I3aeac278b8e1f4bb796e0f3a3f475b3c97208a85
2019-04-04 08:58:04 +02:00
Tobias Henkel cd9827e664
Manage ansible installations within zuul
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
2019-03-15 09:09:16 +01:00
Monty Taylor 522ac60a48
Add line to setup.cfg limiting to python >=3.5
This adds "Requires-Python: >=3.5" to the PKG-INFO file, which will then
in turn be processed by pypi and pip and prevent users on python2 from
being able to pip install zuul.

Change-Id: Ia2fe96bd151faf4f0c15be79872d4ddf9505b771
2018-10-12 14:54:51 -05:00
David Shrewsbury 57ba171f1e Fix pbr setup hook
Helps to name the option correctly.

Change-Id: Iedd0bdda37a6a6e14c327e1a7a914ec554a489d8
2018-10-08 14:53:18 -04:00
James E. Blair da02bdb0b7 Add openstacksdk to executor extras
So that the pbrx image builds install openstacksdk on the executors
so that it can be used by the swift logs role in zuul-jobs.

Change-Id: Ie775169e9230108f7f76ef31953171f5a4881a69
2018-07-27 15:20:16 -07:00
Monty Taylor 975471e4a7
Build container images using pbrx
The pbrx project has a command to build container images based on the
python:alpine base image for any project using pbr and bindep.

Use it to start building container images for zuul. A followup patch
will add support for publishing the images to dockerhub.

The images want the database drivers in them to be useful out of the
box. Also, the executor wants ara installed. By adding those to extra
entries named after the resulting image name, pbrx will install the
extra packages at the right time, as well as produce wheels for them
during the wheel cache stage.

Change-Id: I453b0ab2c4916cb77249b5f1039e6e18fde47ab1
2018-07-16 16:27:46 -05:00
Paul Belanger fc120688f8
Remove zuul-cloner command
With the release of zuul 3.0.0 is command is no longer needed. Jobs
are setup to push the known repo state on to the remove nodes.

Change-Id: I0df6e41dc05276e648d393ec62329a85f1b8c415
Signed-off-by: Paul Belanger <>
2018-04-12 11:49:13 -04:00
Paul Belanger 33a0022cbb
Update pypi metadata
Refresh our metadata for pypi to use our shiny new

Change-Id: I5fdcf86c716f869840265aa40b1046bbb9f1b75e
Signed-off-by: Paul Belanger <>
2018-04-10 13:39:50 -04:00
Paul Belanger 08b63a7b29
Update metadata for pypi
Noticed we haven't updated this information to point to new URL for
documentation or summary.

Change-Id: I4594876888de9aef2259ab9c324fa0b7ed2ebaaa
Signed-off-by: Paul Belanger <>
2018-03-15 20:43:39 -04:00
Monty Taylor 4a781a7f86
Use yarn and webpack to manage zuul-web javascript
yarn drives package and dependency management. webpack handles
bundling, minification and transpiling down to browser-acceptable
javascript but allows for more modern javascript like import statements.

There are some really neat things in the webpack dev server. CSS
changes, for instance, get applied immediately without a refresh. Other
things, like the jquery plugin do need a refresh, but it's handled just
on a file changing.

As a followup, we can also consider turning the majority of the status page
into a webpack library that other people can depend on as a mechanism
for direct use. Things like that haven't been touched because allowing
folks to poke at the existing known status page without too many changes
using the tools seems like a good way for people to learn/understand the

Move things so that the built content gets put
into zuul/web/static so that the built-in static serving from zuul-web
will/can serve the files.

Update so that if npm run build:dist is run before the
python sdist, the built html/javascript content will be
included in the source tarball.

Add a pbr hook so that if yarn is installed, javascript content will be
built before the tarball.

Add a zuul job with a success url that contains a source_url
pointing to the live v3 data.

This adds a framework for verifying that we can serve the web app
urls and their dependencies for all of the various ways we want to
support folks hosting zuul-web.

It includes a very simple reverse proxy server for approximating
what we do in openstack to "white label" the Zuul service -- that
is, hide the multitenancy aspect and present the single tenant
at the site root.

We can run similar tests without the proxy to ensure the default,
multi-tenant view works as well.

Add babel transpiling enabling use of ES6 features

ECMAScript6 has a bunch of nice things, like block scoped variables,
const, template strings and classes. Babel is a javascript transpiler
which webpack can use to allow us to write using modern javascript but
the resulting code to still work on older browsers.

Use the babel-plugin-angularjs-annotate so that angular's dependency
injection doesn't get borked by babel's transpiling things (which causes
variables to otherwise be renamed in a way that causes angular to not
find them)

While we're at it, replace our use of var with let (let is the new
block-scoped version of var) and toss in some use of const and template
strings for good measure.

Add StandardJS eslint config for linting

JavaScript Standard Style is a code style similar to pep8/flake8. It's
being added here not because of the pep8 part, but because the pyflakes
equivalent can catch real errors. This uses the babel-eslint parser
since we're using Babel to transpile already.

This auto-formats the existing code with:

  npm run format

Rather than using StandardJS directly through the 'standard' package,
use the standardjs eslint plugin so that we can ignore the camelCase
rule (and any other rule that might emerge in the future)

Many of under_score/camelCase were fixed in a previous version of the patch.
Since the prevailing zuul style is camelCase methods anyway, those fixes
were left. That warning has now been disabled.

Other things, such as == vs. === and ensuring template
strings are in backticks are fixed.

Ignore indentation errors for now - we'll fix them at the end of this
stack and then remove the exclusion.

Add a 'format' npm run target that will run the eslint command with
--fix for ease of fixing reported issues.

Add a 'lint' npm run target and a 'lint' environment that runs with
linting turned to errors. The next patch makes the lint environment more
broadly useful.

When we run lint, also run the BundleAnalyzerPlugin and set the
success-url to the report.

Add an angular controller for status and stream page

Wrap the status and stream page construction with an angular controller
so that all the javascripts can be bundled in a single file.

Building the files locally is wonderful and all, but what we really want
is to make a tarball that has the built code so that it can be deployed.

Put it in the root source dir so that it can be used with the zuul
fetch-javascript-tarball role.

Also, replace the custom npm job with the new build-javascript-content
job which naturally grabs the content we want.

Make a 'main.js' file that imports the other three so that we just have
a single bundle. Then, add a 'vendor' entry in the common webpack file
and use the CommonsChunkPlugin to extract dependencies into their own
bundle. A second CommonsChunkPlugin entry pulls out a little bit of
metadata that would otherwise cause the main and vendor chunks to change
even with no source change. Then add chunkhash into the filename. This
way the files themselves can be aggressively cached.

This all follows recommendations from and

Change-Id: I2e1230783fe57f1bc3b7818460463df1e659936b
Co-Authored-By: Tristan Cacqueray <>
Co-Authored-By: James E. Blair <>
2018-03-04 07:20:40 -06:00
David Shrewsbury fe1f1944a6 Add finger gateway
This adds the zuul-fingergw app that should be run as root (so that
it can connect to the standard finger port 79), but changes user privs
immediately after binding that port.

Common streaming functions have been moved to to
be shared among modules.

Support for CommandSocket has been included.

Change-Id: Ia35492fe951e7b9367eeab0b145d96189d72c364
2017-12-13 10:07:37 -05:00
Monty Taylor 245a3715db
Add migration tool for v2 to v3 conversion
We need to be able to generate a new v3 config based on our old v2
config. It won't be perfect, but should ultimately be no worse than the
v2.5 auto-generated playbooks.

Change-Id: I9cb676ceff01bbdb845d22774edbaa718323db27
2017-09-10 16:45:29 -06:00
Monty Taylor 51139a0682 Add web-based console log streaming
zuul now provides socket-based console streaming, which is super cool.
In order to have jenkins parity with web streaming, we need to provide a
websocket (javascript in browsers can't really connect to random ports
on servers)

After surveying the existing python websocket options, basically all of
them are based around twisted, eventlet, gevent or asyncio. It's not
just a thing we can easily deal with from our current webob/paste
structure, because it is a change to the fundamental HTTP handling.
While we could write our own websocket server implementation that was
threaded like the rest of zuul, that's a pretty giant amount of work.

Instead, we can run an async-based server that's just for the
websockets, so that we're not all of a sudden putting async code into
the rest of zuul and winding up frankensteined. Since this is new code,
using asyncio and python3 seems like an excellent starting place.

aiohttp supports running a websocket server in a thread. It also
supports doing other HTTP/REST calls, so by going aiohttp we can set
ourselves up for a single answer for the HTTP tier.

In order to keep us from being an open socket relay, we'll expect two
parameters as the first message on the websocket - what's the zuul build
uuid, and what log file do we want to stream. (the second thing,
multiple log files, isn't supported yet by the rest of zuul, but one can
imagine a future where we'd like to support that too, so it's in the
protocol) The websocket server will then ask zuul over gearman for the
IP and port associated with the build and logfile and will start
streaming it to the socket.

Ultimately we'll want the status page to make links of the form:


and we'll want to have apache map the websocket server to something like

Co-Authored-By: Monty Taylor <>

Change-Id: Idd0d3f9259e81fa9a60d7540664ce8d5ad2c298f
2017-07-10 10:32:28 -04:00
Monty Taylor 777c58a461
Stop supporting python2
OpenStack Infra is running Zuul v3 on Python3. We're adding things, like
the websockets streaming, that are python3 only - and testing them turns
out to be extra hard even with skipTest because of the syntax changes.
There are no legacy Zuul v3 installations to worry about, and the
Software Factory team has verified that Python3 on CentOS 7 is not a
problem. It's time to pull the plug.

Remove python2 jobs from running in v3. Make the tox defaults python3
for all testenvs and update setup.cfg to indicate that Zuul v3 supports
python3 and not python2.

Also fixed a python3 bug in zuul command line client that shows up
when running zuul --help under python3.

Change-Id: Icaccab9f6edcdba0b87ec0b5f0411d3dc8030b42
2017-06-16 10:51:03 -05:00
Clint Byrum 5870ccae62 Add support for bwrap
This will be the minimum "batteries included" bubblwrap driver. It does
not do any MAC configuration, since these vary by system. Operators
may wish to wrap it further in a MAC wrapper driver.

Because we set bubblewrap as the default wrapper, test_playbooks tests
it. However, it lacks a negative test, so we won't know if we're not
actually containing things.

Users who don't have bubblewrap or don't wish to use it can set the
untrusted_wrapper to 'nullwrap' which will just execute things as
they're done before this change.

Change-Id: I84dd7c8cc55d2110b58609784007ffda0d135716
Story: 2000910
Task: 3540
Signed-off-by: Paul Belanger <>
2017-06-01 09:26:45 -07:00
Clark Boylan 500992b37d Cleanup docs builds and error on warnings
The docs builds throw quite a few exceptions and warnings. Most of this
is due to code that has moved around so autodoc can't import code to
document that code. Update the names for code that has moved. There were
also so small docstring indentation issues that are corrected.

In addition to fixing these issues, update the minimum version of sphinx
and set warning-is-error so that we avoid these problems in the future
by gating on making sure they don't exist.

Change-Id: Ic06775e7d1c8fbbfc5a8e6c412a2ad15f8699413
2017-04-06 11:49:26 -07:00
Paul Belanger 174a8274d0 Rename zuul-launcher to zuul-executor
To avoid confusion with nodepool-launcher, we've decided to rename
zuul-launcher to zuul-executor.

Change-Id: I7d03cf0f0093400f4ba2e4beb1c92694224a3e8c
Signed-off-by: Paul Belanger <>
2017-03-15 12:21:24 -04:00
Joshua Hesketh 25695cbb51 Merge branch 'master' into feature/zuulv3
Change-Id: I37a3c5d4f12917b111b7eb624f8b68689687ebc4
2017-03-06 09:40:04 -08:00
James E. Blair 5f0f514215 Rename zuul-server zuul-scheduler
This is what we call it.

Change-Id: If4ae79eddff9151979271ae7faf4348014528e46
2017-02-15 15:11:30 -08:00
Joshua Hesketh d78b44878c Add support for sqlalchemy reporter
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.

Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <>
2017-02-06 12:47:29 -05:00
James E. Blair 9f36519297 Add ansible launch server
As part of the Zuulv2.5 effort, this service is not designed to be

Change-Id: Idbac79f60cb694a9a707d204cd274328289bc20d
2016-04-22 09:52:05 -07:00
Jeremy Stanley c3cccc92b6 Replace ci.o.o links with docs.o.o/infra
The documentation site has been deprecated,
replaced by redirects to corresponding paths within where other Project Infrastructure
documentation already resides.

Change-Id: I765c42294c478d88076959462c5316a620a4f24a
2015-05-14 21:38:19 +00:00
Antoine Musso 45dd2cb40e cloner to easily clone dependent repositories
The intent is to replace the devstack shell script with a python utility
which would be easy to reuse.

Change-Id: I3c8748e2544af80e72fdd0b2e3627b4fc2212c01
2014-07-29 09:03:59 +00:00
James E. Blair 4076e2b432 Split the merger into a separate process
Connect it to Zuul via Gearman.  Any number of mergers may be

Directly find the pipeline for a build when processing a result,
so that the procedure is roughly the same for build and merge

The timer trigger currently requires the gerrit trigger also be
configured.  Make that explicit inside of the timer trigger so
that the scheduler API interaction with triggers is cleaner.

Change-Id: I69498813764753c97c426e42d17596c2ef1d87cf
2014-02-17 11:47:15 -08:00
James E. Blair ad28e91698 Add a zuul client
Add a command line client called 'zuul' that supports one command
to start with: 'enqueue'.  It allows an operator (one with access
to the gearman server) to enqueue an arbitrary change in a specified
pipeline.  It uses gearman to communicate with the Zuul server, which
now has an added RPC listener component to answer such requests via

Add tests for the client RPC interface.

Raise an exception if a Gerrit query does not produce a change.  Unlike
events from Gerrit, user (or admin) submitted events over the RPC bus
are more likely to reference invalid changes.  To validate those, the
Gerrit trigger will raise an exception (and remove from its cache) changes
which prove to be invalid.

Change-Id: Ife07683a736c15f4db44a0f9881f3f71b78716b2
2013-12-02 08:32:01 -08:00
Monty Taylor 52905bcdd3 Remove usage of d2to1
d2to1 pulls in setuptools, which trips the unhappy bugs with setuptools
updating. Move past that and just use new pbr.

Change-Id: I2609eda10ed781116940c3607ff85a14fc4b7f58
2013-09-25 15:30:37 -07:00
Paul Belanger feeeaef950 Enable warnerrors for pbr to catch Sphinx warnings
With this change, we now should properly gate on Sphinx errors /

Change-Id: I7e789ad83e81286575c74e1e5256da20a717e982
Signed-off-by: Paul Belanger <>
2013-07-19 21:26:03 -04:00
Monty Taylor bc7588343f Migrate to testrepository.
Needed to move some directory creation around to be contained within the
testcase, but with parallel testing, we shave 48 seconds off the run
time and go from around 60s to around 11. We're also now compatible with
future subunit-based fast-fail semantics when we grow them.

Change-Id: I6c7148c29d1edb5d9469a8c2afe4b31b2b340009
Approved: James E. Blair <>
Reviewed-by: James E. Blair <>
Tested-by: Jenkins
2013-06-27 01:49:59 +00:00
Monty Taylor 5cc8a47d40 Migrate to pbr.
Change-Id: I734f0f8237fb603ee41a39f06e63c007e79825a9
Reviewed-by: James E. Blair <>
Reviewed-by: Doug Hellmann <>
Approved: James E. Blair <>
Tested-by: Jenkins
2013-06-25 19:04:30 +00:00
Clark Boylan 906f8940bd Restrict test coverage to the zuul package.
We are not concerned with the code coverage of upstream libraries that
are used in zuul. Restrict code coverage gathering to the zuul package.

Change-Id: I62eb752c5fd0fe0289253c0b5c950382d760ea95
Approved: James E. Blair <>
Reviewed-by: James E. Blair <>
Tested-by: Jenkins
2012-09-10 20:06:36 +00:00
James E. Blair cdd000764e Add documentation.
Change-Id: I8197ec2e52596fa4136f8af9aa93ea06e56d4d0d
2012-06-08 19:17:28 -07:00