Gerrit has an event plugin for AWS Kinesis (which looks sort of
like Kafka, but without server side checkpoints. or ordering.).
Add support to the Gerrit driver for it for sites which would
rather use that than ssh.
This mocks the /app/installations and /installations/repositories
GitHub API calls to better validate the GitHub project initalization
in the driver.
It implements enough that we can use
GithubClientManager:_prime_installation_map() directly and better
tests the token issuing, etc.
The canMerge check is executed whenever zuul tests if a change can
enter a gate pipeline. This is part of the critical path in the event
handling of the scheduler and therefore must be as fast as
possible. Currently this takes five requests for doing its work and
also transfers large amounts of data that is unneeded:
* get pull request
* get branch protection settings
* get commits
* get status of latest commit
* get check runs of latest commit
Especially when Github is busy this can slow down zuul's event
processing considerably. This can be optimized using graphql to only
query the data we need with a single request. This reduces requests
and load on Github and speeds up event processing in the scheduler.
Since this is the first usage of graphql this also sets up needed
testing infrastructure using graphene to mock the github api with real
Linters are only used by the 'pep8' virtual environment, we do not need
them in other test environment. Also 'pep8' only requires the three
linters it used and the other modules are not needed.
Move the pep8 dependencies to the pep8 testenv.
flake 3.6.0 introduces a couple of new tests, handle them in the zuul
* Disable "W504 line break after binary operator", this is a new warning
with different coding style.
* Fix "F841 local variable 'e' is assigned to but never used"
* Fix "W605 invalid escape sequence" - use raw strings for regexes.
* Fix "F901 'raise NotImplemented' should be 'raise
* Ignore "E252 missing whitespace around parameter equals" since it
reports on parameters like:
def makeNewJobs(self, old_job, parent: Job=None):
Change "flake8: noqa" to "noqa" since "flake8: noqa" is a file level
noqa and gets ignored with flake 3.6.0 if it's not at beginning of line
- this results in many warnings for files ./zuul/driver/bubblewrap/__init__.py and
./zuul/cmd/migrate.py. Fix any issues there.
With the latest cherrypy version the tests run into the hard test
timeout. This is caused by a header conversion issue introduced in
cherrypy 18.0.0. There is a pull request  that fixes this
issue. Once this is merged and released we can uncap it again.
The newest reno release, 2.10.0, requires a feature of newer sphinx without
specifying that the minimum compatibility has changed. Uncap sphinx here
so at least the doc build is not broken, use Sphinx 1.6.1 since that is
the oldest sphinx version supporting what reno needs.
Co-Authored-By: Andreas Jaeger <firstname.lastname@example.org>
Reno v. 2.8.0 added support for customizining the section title
for the 'in-development' release notes. Use that, so our release
notes don't have confusing version numbers like "3.0.0-6".
Some deployments use postgres as database backend. In order to have
official support for that we need to add tests for it.
Mock is part of unittest in python3 and now that zuul is a python3 only
application we can cleanup the package dependency on mock and the
fallback imports that were done to support python2.
The OpenStack Release team has created a great release notes management
tool that integrates with Sphinx. Start using it. For reference on how
to use it, see http://docs.openstack.org/developer/reno/
Add an initial release note with no contents so that the build flow and
docs integration can be verified. The note file can be removed later.
There are several differences between this and testr which we
need to discuss. Let's wait until after the v3 release so we
don't delay it further.
This reverts commit 68e02466ce.
stestr is actively maintained. OpenStack is migrating to stestr,
go ahead and follow suit.
As part of this, shift the default values for STDOUT capture into the
tox.ini file. This preserves the current behavior of tox doing
capture and running tests directly not capturing.
Remove the depend on python-subunit... stestr takes care of that for us.
yarn drives package and dependency management. webpack handles
bundling, minification and transpiling down to browser-acceptable
There are some really neat things in the webpack dev server. CSS
changes, for instance, get applied immediately without a refresh. Other
things, like the jquery plugin do need a refresh, but it's handled just
on a file changing.
As a followup, we can also consider turning the majority of the status page
into a webpack library that other people can depend on as a mechanism
for direct use. Things like that haven't been touched because allowing
folks to poke at the existing known status page without too many changes
using the tools seems like a good way for people to learn/understand the
Move things so that the built content gets put
into zuul/web/static so that the built-in static serving from zuul-web
will/can serve the files.
Update MANIFEST.in so that if npm run build:dist is run before the
included in the source tarball.
built before the tarball.
Add a zuul job with a success url that contains a source_url
pointing to the live v3 data.
This adds a framework for verifying that we can serve the web app
urls and their dependencies for all of the various ways we want to
support folks hosting zuul-web.
It includes a very simple reverse proxy server for approximating
what we do in openstack to "white label" the Zuul service -- that
is, hide the multitenancy aspect and present the single tenant
at the site root.
We can run similar tests without the proxy to ensure the default,
multi-tenant view works as well.
Add babel transpiling enabling use of ES6 features
ECMAScript6 has a bunch of nice things, like block scoped variables,
the resulting code to still work on older browsers.
Use the babel-plugin-angularjs-annotate so that angular's dependency
injection doesn't get borked by babel's transpiling things (which causes
variables to otherwise be renamed in a way that causes angular to not
While we're at it, replace our use of var with let (let is the new
block-scoped version of var) and toss in some use of const and template
strings for good measure.
Add StandardJS eslint config for linting
being added here not because of the pep8 part, but because the pyflakes
equivalent can catch real errors. This uses the babel-eslint parser
since we're using Babel to transpile already.
This auto-formats the existing code with:
npm run format
Rather than using StandardJS directly through the 'standard' package,
use the standardjs eslint plugin so that we can ignore the camelCase
rule (and any other rule that might emerge in the future)
Many of under_score/camelCase were fixed in a previous version of the patch.
Since the prevailing zuul style is camelCase methods anyway, those fixes
were left. That warning has now been disabled.
Other things, such as == vs. === and ensuring template
strings are in backticks are fixed.
Ignore indentation errors for now - we'll fix them at the end of this
stack and then remove the exclusion.
Add a 'format' npm run target that will run the eslint command with
--fix for ease of fixing reported issues.
Add a 'lint' npm run target and a 'lint' environment that runs with
linting turned to errors. The next patch makes the lint environment more
When we run lint, also run the BundleAnalyzerPlugin and set the
success-url to the report.
Add an angular controller for status and stream page
Wrap the status and stream page construction with an angular controller
Building the files locally is wonderful and all, but what we really want
is to make a tarball that has the built code so that it can be deployed.
Put it in the root source dir so that it can be used with the zuul
job which naturally grabs the content we want.
Make a 'main.js' file that imports the other three so that we just have
a single bundle. Then, add a 'vendor' entry in the common webpack file
and use the CommonsChunkPlugin to extract dependencies into their own
bundle. A second CommonsChunkPlugin entry pulls out a little bit of
metadata that would otherwise cause the main and vendor chunks to change
even with no source change. Then add chunkhash into the filename. This
way the files themselves can be aggressively cached.
This all follows recommendations from https://webpack.js.org/guides/caching/https://webpack.js.org/guides/code-splitting/ and
Co-Authored-By: Tristan Cacqueray <email@example.com>
Co-Authored-By: James E. Blair <firstname.lastname@example.org>
Since we have some python3 typehints in the code now, there is no need
to duplicate that information in sphinx docstrings.
sphinx-autodoc-typehints reads type annotations and updates :type:
information for :param: and :rtype: for :returns:. The results are the
same as if :param: had been given a type or a :type: directive, so this
is essentially just in service of using the same information for both
type hints/static analysis and for documentation.
There is a bug in the released version in that it does not consider
someone setting a default domain explicitly such as we do in zuul. A
pull request has been submitted that fixes the issue
This renames the internal sphinx domain from zuul to zuuldoc. This
is a temporary measure to avoid colliding with the domain defined
in zuul-sphinx (because of the initial data required, it's not
easy for us to simply add new directives/roles without at least
porting *some* of the framework into zuul-sphinx).
I expect to do that when this has stabilized.
We need to be able to link to configuration attributes, and they
should show up in the index. To that end, add some sphinx
directives to support config objects and attributes. These handle
nesting so that when we get deep into nested yaml (eg,
pipeline.trigger.gerrit.event) the full path will appear in the
header for the attribute. The ancestors will not be as prominent.
This ends up looking like the python class.FUNCTION() headers
in the stdlib docs.
The implementation is based on, and is compatible with, the nascent
zuul-sphinx module. Once that is published, we can either move
this code into that module, or depend on that module and add these
directives to the domain it creates.
The sphinx theme is changed to the current Sphinx default. That
is the theme "alabaster" (note, this is distinct from the theme
named "default", which is the old python2 style theme). Alabaster
has top-notch typography, and most importantly, it renders the
kinds of nested descriptors we're using very well.
python3 includes support for optional type annotations which can be used by
static analysis tools to perform type checking. The mypy tool is a
static type checking tool that can also infer type information in many
cases, but which will use explicit type information if it is present.
Add mypy to test-requirements and to the pep8 job so that our pep8 job
can do more analysis work and less with the code style.
To support this, there were a few places in the current codebase that
needed an explicit type hint. For variables/attributes in 3.5 this is done via
comments. There is a conditional import that was confusion that just got
marked with an 'ignore'.
Our ansible action and lookup plugins confuse mypi with the way they
import the ansible base classes. That's ok - they confuse us with that
too. The .pyi files are 'typeshed' files, which are a way that one can
provide static type annotations without putting the information into the
file itself. mypy will always prefer a .pyi file over a .py file (since
the point of them is to be external annotion/interface description) So
in order to get mypy to not barf on the ansible import weirdness, just
add a corresponding empty .pyi file. We could potentially actually put
interface descriptions in them - but I don't think there is very much
value in that.
It should be amusing to at least someone that we have to flake8: noqa
an import from typing that was done to provide a type hint in a comment.
Current hacking library pins on a version of flake8 that is too
old to recognize some py3.5'isms.
Adds ignores for:
E305 expected 2 blank lines after class or function definition
E402 module level import not at top of file
F405 'AnsibleModule' may be undefined, or defined from star imports
W503 line break before binary operator
The docs builds throw quite a few exceptions and warnings. Most of this
is due to code that has moved around so autodoc can't import code to
document that code. Update the names for code that has moved. There were
also so small docstring indentation issues that are corrected.
In addition to fixing these issues, update the minimum version of sphinx
and set warning-is-error so that we avoid these problems in the future
by gating on making sure they don't exist.
The zuul v3 architecture does not require that this be supported
as an internal special case. Remove it for now in the expectation
that we will implement similar functionality using the auth system
and Ansible roles.
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.
Signed-off-by: Paul Belanger <email@example.com>
Rely on blockdiag to illustrate the CRD examples.
Change some references to "check" and "gate" to respectively
"Independent pipeline" and "Dependent pipeline". Since third parties
might use a different terminology.
Decorate note with '.. note::'
To render blockdiag nodes as circles, we need blockdiag 1.1.0. Bump the
test requirements for sphinxcontrib-blockdiag to at least 1.1.0.
This was added to work around bug 1091333. A new sphinx has been
released and so this is no longer necessary.
Also update sphinx requirement with global requirements.
Since keystoneclient pulls in lxml which needs various system
library headers to build its bindings, and swiftclient depends on
keystoneclient, and zuul handles these as optional dependencies,
make them test requirements.
According to https://wiki.openstack.org/wiki/Python3 dict.iteritems()
should be replaced with six.iteritems(dict).
Added six >= 1.6.0 as new requirement. Removed six from test requirements.
ASCII art is fun but not savvy for project managers and directors. This
patch slightly enhance the 'gating' documentation with colored diagrams.
This is made possible via http://blockdiag.com/ by Takeshi KOMIYA who
even took the time to write a sphinx extension. The version dependency
is at the very least 0.5.5, but might be higher :/
Needed to move some directory creation around to be contained within the
testcase, but with parallel testing, we shave 48 seconds off the run
time and go from around 60s to around 11. We're also now compatible with
future subunit-based fast-fail semantics when we grow them.
Approved: James E. Blair <firstname.lastname@example.org>
Reviewed-by: James E. Blair <email@example.com>
Reviewed-by: James E. Blair <firstname.lastname@example.org>
Approved: Jeremy Stanley <email@example.com>
Reviewed-by: Jeremy Stanley <firstname.lastname@example.org>