Commit Graph

49 Commits (master)

Author SHA1 Message Date
James E. Blair e6e978615f Add AWS Kinesis support
Gerrit has an event plugin for AWS Kinesis (which looks sort of
like Kafka, but without server side checkpoints.  or ordering.).

Add support to the Gerrit driver for it for sites which would
rather use that than ssh.

Change-Id: I942845ac16bf220664499f14ff7c4086ff65de2a
2023-07-25 11:04:19 -07:00
Ian Wienand 403d1c2882 github: more complete mocking for app setup
This mocks the /app/installations and /installations/repositories
GitHub API calls to better validate the GitHub project initalization
in the driver.

It implements enough that we can use
GithubClientManager:_prime_installation_map() directly and better
tests the token issuing, etc.

Change-Id: I608f1540ef33b1a95595393e546afba308fef66a
2021-09-23 19:53:48 +10:00
Tobias Henkel 4c972f00bd
Optimize canMerge using graphql
The canMerge check is executed whenever zuul tests if a change can
enter a gate pipeline. This is part of the critical path in the event
handling of the scheduler and therefore must be as fast as
possible. Currently this takes five requests for doing its work and
also transfers large amounts of data that is unneeded:

* get pull request
* get branch protection settings
* get commits
* get status of latest commit
* get check runs of latest commit

Especially when Github is busy this can slow down zuul's event
processing considerably. This can be optimized using graphql to only
query the data we need with a single request. This reduces requests
and load on Github and speeds up event processing in the scheduler.

Since this is the first usage of graphql this also sets up needed
testing infrastructure using graphene to mock the github api with real
test data.

Change-Id: I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10
2020-02-28 09:43:56 +01:00
Antoine Musso 73110b4909 tox: reduce deps used for pep8 env
Linters are only used by the 'pep8' virtual environment, we do not need
them in other test environment.  Also 'pep8' only requires the three
linters it used and the other modules are not needed.

Move the pep8 dependencies to the pep8 testenv.

Change-Id: I3a8436cbff9e2386cf0e0f5b0d1742283e647b56
2020-01-21 16:05:18 +01:00
Tristan Cacqueray 85616c4c09 web: add OpenAPI documentation
This change adds a swagger description of the REST API. The description is
rendered in the sphinx user documentations and in the web interface.

Change-Id: I753524f40a09874dab5952f14ab17025525bbab9
2019-06-12 22:35:13 +02:00
James E. Blair 5f3a435c14 Cap mypy
The 0.650 release appears to dislike our use of pyyaml.

Change-Id: I4ac0029b9b84d523ad5a09293886d3b7a83a3175
2018-12-07 15:00:17 -08:00
Zuul 5f7f7eaf40 Merge "Use doc/requirements.txt" 2018-10-29 07:11:37 +00:00
Andreas Jaeger cb9e1b4885 Use doc/requirements.txt
Create separate doc/requirements.txt file and use it, this shrinks
the list of requirements to install for a normal run.

Change-Id: I7c27f7f737657005ab6e370ae7e5dceb5a7073e7
2018-10-28 16:59:15 +01:00
Andreas Jaeger d9059524e0 Fix flake 3.6.0 warnings
flake 3.6.0 introduces a couple of new tests, handle them in the zuul

* Disable "W504 line break after binary operator", this is a new warning
  with different coding style.
* Fix "F841 local variable 'e' is assigned to but never used"
* Fix "W605 invalid escape sequence" - use raw strings for regexes.
* Fix "F901 'raise NotImplemented' should be 'raise
* Ignore "E252 missing whitespace around parameter equals" since it
  reports on parameters like:
  def makeNewJobs(self, old_job, parent: Job=None):

Change "flake8: noqa" to "noqa" since "flake8: noqa" is a file level
noqa and gets ignored with flake 3.6.0 if it's not at beginning of line
- this results in many warnings for files ./zuul/driver/bubblewrap/ and
./zuul/cmd/ Fix any issues there.

Change-Id: Ia79bbc8ac0cd8e4819f61bda0091f4398464c5dc
2018-10-28 16:39:30 +01:00
James E. Blair b27757c274 Pin flake8
A new release behaves... differently.

Change-Id: Ia9fb2986f84942c7ea961366c1f5891409b420dc
2018-10-24 11:18:17 -07:00
Tobias Henkel 85f6598722 Cap cherrypy and uncap sphinx
With the latest cherrypy version the tests run into the hard test
timeout. This is caused by a header conversion issue introduced in
cherrypy 18.0.0. There is a pull request [1] that fixes this
issue. Once this is merged and released we can uncap it again.

The newest reno release, 2.10.0, requires a feature of newer sphinx without
specifying that the minimum compatibility has changed. Uncap sphinx here
so at least the doc build is not broken, use Sphinx 1.6.1 since that is
the oldest sphinx version supporting what reno needs.


Change-Id: Ic00e664152062d1fc1677e467b638ac2be99aef1
Co-Authored-By: Andreas Jaeger <>
2018-09-04 15:40:08 -07:00
James E. Blair ef23bd4862 Revert "Revert "Switch to stestr""
This reverts commit e56801f2e8.

Change-Id: Icbea5ddd744d1fa65a397fb613731315687f9803
2018-05-17 08:33:40 -07:00
Zuul 025477318b Merge "Update in-development releasenotes title" 2018-04-10 18:28:24 +00:00
James E. Blair 4fc6d5dcf4 Update in-development releasenotes title
Reno v. 2.8.0 added support for customizining the section title
for the 'in-development' release notes.  Use that, so our release
notes don't have confusing version numbers like "3.0.0-6".

Change-Id: I604e8e4608b3c2a244fd15e020129619fc4fd2d4
2018-04-10 08:39:14 -07:00
Tobias Henkel 267d134b7e
Add tests for postgresql
Some deployments use postgres as database backend. In order to have
official support for that we need to add tests for it.

Change-Id: I36408c27ccb36fd0826d97f34d98b211ee075f4e
2018-04-10 15:51:25 +02:00
Clark Boylan e0eea30dbf Cleanup use of mock package in for python2
Mock is part of unittest in python3 and now that zuul is a python3 only
application we can cleanup the package dependency on mock and the
fallback imports that were done to support python2.

Change-Id: I55a9831b96bdc384ff04723b16c8af42084af03c
2018-03-29 18:10:23 -07:00
Monty Taylor f0314feb8c
Use reno for release notes
The OpenStack Release team has created a great release notes management
tool that integrates with Sphinx. Start using it. For reference on how
to use it, see

Add an initial release note with no contents so that the build flow and
docs integration can be verified. The note file can be removed later.

Change-Id: I254cd220fc8c0c06ee87f84f1fb5cbe3244f0fed
2018-03-28 15:52:21 -05:00
James E. Blair e56801f2e8 Revert "Switch to stestr"
There are several differences between this and testr which we
need to discuss.  Let's wait until after the v3 release so we
don't delay it further.

This reverts commit 68e02466ce.

Change-Id: I873145c15fd88fe752cb987d0892c146c35041d8
2018-03-21 07:50:08 -07:00
Monty Taylor 68e02466ce
Switch to stestr
stestr is actively maintained. OpenStack is migrating to stestr,
go ahead and follow suit.

As part of this, shift the default values for STDOUT capture into the
tox.ini file. This preserves the current behavior of tox doing
capture and running tests directly not capturing.

Remove the depend on python-subunit... stestr takes care of that for us.

Change-Id: I90ae343a5db966f74e888bb98aab0bf39dcdd2cb
2018-03-18 10:41:40 -05:00
Monty Taylor 4a781a7f86
Use yarn and webpack to manage zuul-web javascript
yarn drives package and dependency management. webpack handles
bundling, minification and transpiling down to browser-acceptable
javascript but allows for more modern javascript like import statements.

There are some really neat things in the webpack dev server. CSS
changes, for instance, get applied immediately without a refresh. Other
things, like the jquery plugin do need a refresh, but it's handled just
on a file changing.

As a followup, we can also consider turning the majority of the status page
into a webpack library that other people can depend on as a mechanism
for direct use. Things like that haven't been touched because allowing
folks to poke at the existing known status page without too many changes
using the tools seems like a good way for people to learn/understand the

Move things so that the built content gets put
into zuul/web/static so that the built-in static serving from zuul-web
will/can serve the files.

Update so that if npm run build:dist is run before the
python sdist, the built html/javascript content will be
included in the source tarball.

Add a pbr hook so that if yarn is installed, javascript content will be
built before the tarball.

Add a zuul job with a success url that contains a source_url
pointing to the live v3 data.

This adds a framework for verifying that we can serve the web app
urls and their dependencies for all of the various ways we want to
support folks hosting zuul-web.

It includes a very simple reverse proxy server for approximating
what we do in openstack to "white label" the Zuul service -- that
is, hide the multitenancy aspect and present the single tenant
at the site root.

We can run similar tests without the proxy to ensure the default,
multi-tenant view works as well.

Add babel transpiling enabling use of ES6 features

ECMAScript6 has a bunch of nice things, like block scoped variables,
const, template strings and classes. Babel is a javascript transpiler
which webpack can use to allow us to write using modern javascript but
the resulting code to still work on older browsers.

Use the babel-plugin-angularjs-annotate so that angular's dependency
injection doesn't get borked by babel's transpiling things (which causes
variables to otherwise be renamed in a way that causes angular to not
find them)

While we're at it, replace our use of var with let (let is the new
block-scoped version of var) and toss in some use of const and template
strings for good measure.

Add StandardJS eslint config for linting

JavaScript Standard Style is a code style similar to pep8/flake8. It's
being added here not because of the pep8 part, but because the pyflakes
equivalent can catch real errors. This uses the babel-eslint parser
since we're using Babel to transpile already.

This auto-formats the existing code with:

  npm run format

Rather than using StandardJS directly through the 'standard' package,
use the standardjs eslint plugin so that we can ignore the camelCase
rule (and any other rule that might emerge in the future)

Many of under_score/camelCase were fixed in a previous version of the patch.
Since the prevailing zuul style is camelCase methods anyway, those fixes
were left. That warning has now been disabled.

Other things, such as == vs. === and ensuring template
strings are in backticks are fixed.

Ignore indentation errors for now - we'll fix them at the end of this
stack and then remove the exclusion.

Add a 'format' npm run target that will run the eslint command with
--fix for ease of fixing reported issues.

Add a 'lint' npm run target and a 'lint' environment that runs with
linting turned to errors. The next patch makes the lint environment more
broadly useful.

When we run lint, also run the BundleAnalyzerPlugin and set the
success-url to the report.

Add an angular controller for status and stream page

Wrap the status and stream page construction with an angular controller
so that all the javascripts can be bundled in a single file.

Building the files locally is wonderful and all, but what we really want
is to make a tarball that has the built code so that it can be deployed.

Put it in the root source dir so that it can be used with the zuul
fetch-javascript-tarball role.

Also, replace the custom npm job with the new build-javascript-content
job which naturally grabs the content we want.

Make a 'main.js' file that imports the other three so that we just have
a single bundle. Then, add a 'vendor' entry in the common webpack file
and use the CommonsChunkPlugin to extract dependencies into their own
bundle. A second CommonsChunkPlugin entry pulls out a little bit of
metadata that would otherwise cause the main and vendor chunks to change
even with no source change. Then add chunkhash into the filename. This
way the files themselves can be aggressively cached.

This all follows recommendations from and

Change-Id: I2e1230783fe57f1bc3b7818460463df1e659936b
Co-Authored-By: Tristan Cacqueray <>
Co-Authored-By: James E. Blair <>
2018-03-04 07:20:40 -06:00
Monty Taylor 12696f225d
Remove pep8 and pyflakes from test-requirements
They are transitive depends of flake8 already.

Change-Id: I3a73e4a42ae300d97e175f914c3c1166b680117b
2018-01-23 10:11:57 -06:00
Monty Taylor a0f4f170c0
Remove python-keystoneclient from test-requirements
We don't need it for any purpose.

Change-Id: Ib8e6039fc72f9f813eabf15e8ef7b216215576da
2018-01-23 10:11:57 -06:00
Monty Taylor fb95d09b36
Add sphinx-autodoc-typehits sphinx extension
Since we have some python3 typehints in the code now, there is no need
to duplicate that information in sphinx docstrings.
sphinx-autodoc-typehints reads type annotations and updates :type:
information for :param: and :rtype: for :returns:. The results are the
same as if :param: had been given a type or a :type: directive, so this
is essentially just in service of using the same information for both
type hints/static analysis and for documentation.

There is a bug in the released version in that it does not consider
someone setting a default domain explicitly such as we do in zuul. A
pull request has been submitted that fixes the issue

Change-Id: I900a58eb6503cfee3cdff83e6cc376b5e2da3d44
2017-08-10 09:45:54 -05:00
James E. Blair bbaf222aa4 Add zuul-sphinx as a requirement
This renames the internal sphinx domain from zuul to zuuldoc.  This
is a temporary measure to avoid colliding with the domain defined
in zuul-sphinx (because of the initial data required, it's not
easy for us to simply add new directives/roles without at least
porting *some* of the framework into zuul-sphinx).

I expect to do that when this has stabilized.

Change-Id: I26be542a6d4f6266f7843ada5939172656b8b847
Depends-On: I8a1534f7c2614ee11411cf228de38931257fc970
2017-08-07 14:56:17 -07:00
James E. Blair 1761e8623f Use default sphinx theme and index attributes
We need to be able to link to configuration attributes, and they
should show up in the index.  To that end, add some sphinx
directives to support config objects and attributes.  These handle
nesting so that when we get deep into nested yaml (eg,
pipeline.trigger.gerrit.event) the full path will appear in the
header for the attribute.  The ancestors will not be as prominent.
This ends up looking like the python class.FUNCTION() headers
in the stdlib docs.

The implementation is based on, and is compatible with, the nascent
zuul-sphinx module.  Once that is published, we can either move
this code into that module, or depend on that module and add these
directives to the domain it creates.

The sphinx theme is changed to the current Sphinx default.  That
is the theme "alabaster" (note, this is distinct from the theme
named "default", which is the old python2 style theme).  Alabaster
has top-notch typography, and most importantly, it renders the
kinds of nested descriptors we're using very well.

Change-Id: I673b20849dd808e8fbff33fa1a7524227d1a6011
2017-07-28 10:37:12 -07:00
Monty Taylor fb8f5a44bd
Use mypy to do static type checking
python3 includes support for optional type annotations which can be used by
static analysis tools to perform type checking. The mypy tool is a
static type checking tool that can also infer type information in many
cases, but which will use explicit type information if it is present.

Add mypy to test-requirements and to the pep8 job so that our pep8 job
can do more analysis work and less with the code style.

To support this, there were a few places in the current codebase that
needed an explicit type hint. For variables/attributes in 3.5 this is done via
comments. There is a conditional import that was confusion that just got
marked with an 'ignore'.

Our ansible action and lookup plugins confuse mypi with the way they
import the ansible base classes. That's ok - they confuse us with that
too. The .pyi files are 'typeshed' files, which are a way that one can
provide static type annotations without putting the information into the
file itself. mypy will always prefer a .pyi file over a .py file (since
the point of them is to be external annotion/interface description) So
in order to get mypy to not barf on the ansible import weirdness, just
add a corresponding empty .pyi file. We could potentially actually put
interface descriptions in them - but I don't think there is very much
value in that.

It should be amusing to at least someone that we have to flake8: noqa
an import from typing that was done to provide a type hint in a comment.

Change-Id: I6c4ac3dcfc6fd990e6c6886749de147ad28389d1
2017-07-27 14:34:07 -05:00
James E. Blair 157df0279e Use oslosphinx theme
It looks nicer than the default, and many of our other docs use it.

Change-Id: I907518e9c86655c7be91996e246e35d01b7ef393
2017-07-05 14:38:31 -07:00
David Shrewsbury 00457feefd Remove use of hacking lib
Current hacking library pins on a version of flake8 that is too
old to recognize some py3.5'isms.

Adds ignores for:
  E305 expected 2 blank lines after class or function definition
  E402 module level import not at top of file
  F405 'AnsibleModule' may be undefined, or defined from star imports
  W503 line break before binary operator

Change-Id: I30ae2b6745c8696c58ef1d57790b4268c22cd356
2017-05-18 14:23:55 -04:00
Monty Taylor 88187074c3
Block sphinx 1.6.1
There are issues with the sphinx 1.6 release. While they're being sorted
out, block it from test-requirements.txt.

Change-Id: If451c5b6e4634b1695bcbfbb6889cd828876c258
2017-05-16 16:15:39 -05:00
Clark Boylan 500992b37d Cleanup docs builds and error on warnings
The docs builds throw quite a few exceptions and warnings. Most of this
is due to code that has moved around so autodoc can't import code to
document that code. Update the names for code that has moved. There were
also so small docstring indentation issues that are corrected.

In addition to fixing these issues, update the minimum version of sphinx
and set warning-is-error so that we avoid these problems in the future
by gating on making sure they don't exist.

Change-Id: Ic06775e7d1c8fbbfc5a8e6c412a2ad15f8699413
2017-04-06 11:49:26 -07:00
James E. Blair 92e953a807 Remove internal Swift support
The zuul v3 architecture does not require that this be supported
as an internal special case.  Remove it for now in the expectation
that we will implement similar functionality using the auth system
and Ansible roles.

Change-Id: I9c1e05a716ce8cfb5f66c4688db110c320d33b4b
2017-03-07 13:15:02 -08:00
Joshua Hesketh 25695cbb51 Merge branch 'master' into feature/zuulv3
Change-Id: I37a3c5d4f12917b111b7eb624f8b68689687ebc4
2017-03-06 09:40:04 -08:00
Paul Belanger 7641d80462 Update hacking dependency
Sync with openstack global requirements for hacking. This allows us to
properly use pbr 2.0.0.

Change-Id: Ia20a1d743e34641e1ff1719e8527f1ee67c8ab3e
Signed-off-by: Paul Belanger <>
2017-03-01 13:22:44 -05:00
Joshua Hesketh d78b44878c Add support for sqlalchemy reporter
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.

Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <>
2017-02-06 12:47:29 -05:00
Swapnil Kulkarni (coolsvap) dab2fafe8a Remove discover from test-requirements
It's only needed for python < 2.7 which is not supported

Change-Id: I370b3fad3cc926ffb600ca669745d11cb1a5a621
2016-07-22 05:07:08 +00:00
Antoine Musso 281a89ebf5 Enhance CRD doc with diagrams
Rely on blockdiag to illustrate the CRD examples.

Change some references to "check" and "gate" to respectively
"Independent pipeline" and "Dependent pipeline". Since third parties
might use a different terminology.

Decorate note with '.. note::'

To render blockdiag nodes as circles, we need blockdiag 1.1.0. Bump the
test requirements for sphinxcontrib-blockdiag to at least 1.1.0.

Change-Id: I4b0600309bbc3bc238c80c5ea78d9d3c233b6ef2
2015-11-18 15:19:51 +11:00
Tobias Henkel dd78ef2dcc Support pagination in simplequery for Gerrit >=2.9
Starting with Gerrit 2.9 the pagination mechanism changed to using
offsets instead of sortkeys. For details please check the release
notes of 2.9:

In order to support both versions it is necessary to check for the
presence of moreChanges in the status line and then process the next
chunks either with the old or the new method.

Change-Id: Ieb54052debd07d86db66ef0f1d7ce1bdb42964b4
2015-08-31 11:42:15 +02:00
Andreas Jaeger 43f5a773e6 Remove docutils pin
This was added to work around bug 1091333.  A new sphinx has been
released and so this is no longer necessary.

Also update sphinx requirement with global requirements.

Change-Id: I9107884d964757a9154bdd1feb4ea6df4e9df43e
2014-10-14 20:32:02 +02:00
Jeremy Stanley f988eea62a Make keystone and swift clients optional test reqs
Since keystoneclient pulls in lxml which needs various system
library headers to build its bindings, and swiftclient depends on
keystoneclient, and zuul handles these as optional dependencies,
make them test requirements.

Change-Id: I5c7db10644dcf2810cbfdfe8c11d4790909c04cf
2014-09-10 22:27:33 +00:00
Christian Berendt 145d9d7ae6 Bump hacking to 0.9.x series
Change-Id: Ic21d2f72a073575db40500e38f2bd59e68d95dfa
2014-07-20 19:19:57 +02:00
Christian Berendt e35b99cb5c replace dict.iteritems() with six.iteritems(dict)
According to dict.iteritems()
should be replaced with six.iteritems(dict).

Added six >= 1.6.0 as new requirement. Removed six from test requirements.

Change-Id: Ifba282b1bde9681268e6e46ffd024c96c6b9be05
2014-06-06 17:10:44 +02:00
tanlin e621a417b4 Use six.moves.urllib.parse instead of urlparse
To keep Python 3.x compatibility, use six.moves.urllib.parse
to replace urlparse.

Add six as a dependency in test_requirements.txt

Change-Id: Ia6cd042970c2d4e5d418a39e25bab32b4d93c544
2014-02-25 11:44:25 +08:00
James E. Blair ec934d914b Pin Sphinx to <1.2
1.2 was just released and has a bug that causes builds to fail with
the error:

  error: 'source_dir' must be a directory name (got `...doc/source`)


Change-Id: I53b49cdc00851f0bddfdd18d0cf4e095a917aaa4
2013-12-10 08:41:18 -08:00
James E. Blair 4ce47daa50 Document zuul client
Adds programoutput sphinx extension as a test dependency so doc
builds can include the program help text.

Change-Id: Iec2f09f710162614cbb393a5628204ddebe2e29f
2013-12-05 14:06:17 -08:00
Antoine Musso 3a43e14bda Doc: enhance gating reference with diagrams
ASCII art is fun but not savvy for project managers and directors. This
patch slightly enhance the 'gating' documentation with colored diagrams.

This is made possible via by Takeshi KOMIYA who
even took the time to write a sphinx extension.  The version dependency
is at the very least 0.5.5, but might be higher :/

Change-Id: Ibe3c2674a5dff2114c40a84ffdec8a8886b1b21b
2013-11-07 17:34:07 +01:00
Monty Taylor c779550fac Partial sync with OpenStack requirements.
Where we're using the same libraries as OpenStack, sync with the
OpenStack versions. Just to be nice.

Change-Id: I8e90d2a8945d62e962b813c6396f0e7db4e14222
2013-09-25 15:30:37 -07:00
Clark Boylan 95b541c293 Include discover in test requirements.
* test-requirements.txt: Discover is needed for python26 tests.

Change-Id: If467058a326b285f394b63acf3432a98213f24fa
2013-07-18 14:19:20 -07:00
Monty Taylor bc7588343f Migrate to testrepository.
Needed to move some directory creation around to be contained within the
testcase, but with parallel testing, we shave 48 seconds off the run
time and go from around 60s to around 11. We're also now compatible with
future subunit-based fast-fail semantics when we grow them.

Change-Id: I6c7148c29d1edb5d9469a8c2afe4b31b2b340009
Approved: James E. Blair <>
Reviewed-by: James E. Blair <>
Tested-by: Jenkins
2013-06-27 01:49:59 +00:00
Monty Taylor 4998315d03 Rename requirements files to standard names.
Change-Id: I0385ac9f8336a8c2e45b0fa6ed062b6b23a182aa
Reviewed-by: James E. Blair <>
Approved: Jeremy Stanley <>
Reviewed-by: Jeremy Stanley <>
Tested-by: Jenkins
2013-06-25 21:52:17 +00:00