An SQL database is an additional requirement so we should also include
the client libraries for it.

Change-Id: Id3a8b55928d9ce9e8764f18a64a42bb2f14f5ecb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

The release of pyjwt 2.0.0 changed the behavior of some functions, which
caused errors. Fix the errors, use pyjwt 2.0.0's better handling of JWKS,
and pin requirement to 2.X to avoid future potential API breaking changes.

Change-Id: Ibef736e0f635dfaf4477cc2a90a22665da9f1959

It has the capability to index build and buildset results.

With the help of tools like Kibana, advanced analytics
dashboard could be built on top of the Zuul Elasticsearch
index.

Optionally job's variables and zuul_return data can be
exported along with build results under the job_vars and
job_returned_vars fields.

Change-Id: I5315483c55c10de63a3cd995ef681d0b64b98513

Removes outdated workaround on importlib-resources.

This fixes issue with py39+, where stdlib already provides it.

Change-Id: I82264c293ff9072e970e81417658fc934cccea2d

Release 2.0.0 on December 22, 2020 brought breaking changes. Pin to the
last working version until we can fix the breaks properly.

Change-Id: I2e6cc36d373f8c774eda5f5caec9868f74023729

This reverts commit a58b29c6bf.  The
dependency is uncapped in zuul-client now.

Change-Id: I9d31326c444cf225442c0b46e040312be9a2ff73

The reasoning is explained inline.  We can remove this when
zuul-client is fixed.

Change-Id: I431d9202cabcc9bf7b60fd643c55b557f6505185

Branch protection rules in github are fn patterns which are currently
matched locally in zuul. This is error prone and can lead in edge
cases to wrong matches resulting in wrong enqueue decisions into gate
pipelines. When requesting branch protection rules in github we also
can request the matching refs along with the rules. This is much safer
since we can plain text match them against the change branch.

Change-Id: Ic995d4b2e16a5d741f0209fa9236959d8f4d10b9

These various exceptions are raised as certain parts of the
configuration are parsed.  Although they give you a lot of information
about the issue, they don't give you any context for what part of your
configuration caused the error to be raised.

They are all working on a particular item of the configuration that I
think it would be very useful to show in the error output.

This was inspired by me putting playbooks in the .zuul.d directory,
where they were read as config files.  In hindsight it was obvious,
but it took me a while to figure out what was going on.  With this it
would have reported something like:

  Zuul encountered a syntax error while parsing its configuration in
  the repo org/project1 on branch master.  The error was:

  Configuration item has more than one key.  Each zuul.yaml
  configuration file must be a list of dictionaries with a
  single key, for example:

  - job:
      name: foo
  ... blah ...

  The incorrect values are:

  hosts: all
  tasks:
  - debug:
      msg: blah
    name: foo

which would have clued me in immediately.

This truncates the message if it is too long; which might be the case
if dumped a playbook or foreign config file.  However I think
practically most errors are typo level on jobs/templates/etc. that
will show nicely.

The YAML version is updated to support non-sorted dumping of the keys.
This is particularly helpful with Python 3.7+ and default ordered
dicts.

Change-Id: I4851a5a796fa452a023e0e6193fc724ae0967a44

This has a fix required for using TLS.

Change-Id: I6d29ca3bf31b1f50cebc245d3db2095ab65921c4

This has a fix required for using TLS.

Change-Id: I6d29ca3bf31b1f50cebc245d3db2095ab65921c4

Cheroot 8.3.1 have been released with the https://github.com/cherrypy/cheroot/issues/263
fix.

Change-Id: I83aad30a6145ed2c7689d6c25c624c453c07ed2b

Zuul is using a dateutil provided transitively. This change
makes the requirements explicit.

Change-Id: I672d409a1bb241e5d199c77acdd51a8260d99ee2

Current set of dependencies brings latest cherrypy which is
incompatible with cheroot we need. This pins cherrypy to last
version that is compatible. The current verision of pip will
still install cherrypy, but the upcoming depsolver is unable to
resolve the conflict.

We also add a fail-safe `pip check` that runs the new depsolver
and should prevent a similar regressions from appearing.

Change-Id: I2b0fa43b906d0919854d30da0ee825002444b201

I installed Zuul 18.0 on a CentOS 7 with Python 3.6 from RH SCL repos
which comes with virtualenv==15.1.0. This is effectively too old to
provision the virtual environments for Ansible, likely because that
version of virtualenv hardcodes PIP 9. During the installation,
zuul-executor dies with this error:

 Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, six, pynacl, cryptography, bcrypt, paramiko, ansible, XStatic-Bootstrap-SCSS, pyfakefs, XStatic-Patternfly-Bootstrap-Treeview, wcwidth, pyparsing, pyperclip, cmd2, pbr, stevedore, PrettyTable, cliff, XStatic, XStatic-Patternfly, wrapt, debtcollector, msgpack, pytz, netaddr, iso8601, netifaces, Babel, oslo.i18n, oslo.utils, oslo.serialization, itsdangerous, Werkzeug, click, Flask, SQLAlchemy, Flask-SQLAlchemy, XStatic-jQuery, Mako, python-editor, python-dateutil, alembic, Flask-Migrate, pygments, junit-xml, Flask-Script, Frozen-Flask, extras, linecache2, traceback2, argparse, unittest2, fixtures, python-mimeparse, testtools, python-subunit, XStatic-DataTables, ara, os-service-types, urllib3, chardet, idna, certifi, requests, keystoneauth1, jsonpointer, jsonpatch, appdirs, munch, decorator, jmespath, requestsexceptions, dogpile.cache, openstacksdk, dictdiffer, websocket-client, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, oauthlib, requests-oauthlib, kubernetes, python-string-utils, ruamel.yaml.clib, ruamel.yaml, openshift, google-resumable-media, protobuf, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-storage
 Running setup.py install for PyYAML: started
 Running setup.py install for PyYAML: finished with status 'done'
 Running setup.py install for ansible: started
 Running setup.py install for ansible: finished with status 'done'
 Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: started
 Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: finished with status 'error'
 Complete output from command /var/lib/zuul/ansible-bin/2.6/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-24z_rrjk/XStatic-Patternfly-Bootstrap-Treeview/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6ie_p40b-record/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/zuul/ansible-bin/2.6/include/site/python3.6/XStatic-Patternfly-Bootstrap-Treeview:
 Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/__init__.py", line 10, in <module>
 from setuptools.extern.six.moves import filter, filterfalse, map
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/extern/__init__.py", line 1, in <module>
 from pkg_resources.extern import VendorImporter
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3017, in <module>
 @_call_aside
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
 f(*args, **kwargs)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set
 dist.activate(replace=False)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2577, in activate
 declare_namespace(pkg)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2151, in declare_namespace
 _handle_ns(packageName, path_item)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2091, in _handle_ns
 _rebuild_mod_path(path, packageName, module)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path
 orig_path.sort(key=position_in_sys_path)
 AttributeError: '_NamespacePath' object has no attribute 'sort'

This points to https://github.com/pypa/pip/issues/4216, so the fix is
apparently to make sure that there's new enough PIP + setuptools in the
created virtual env, and these versions are determined by host's
virtualenv module (thanks to webknjaz on IRC for explaining this to me).

After a restart zuul-executor runs normally, but as a side effect not
all required Python modules are installed for Ansible's usage. For me,
this meant that (e.g.) `upload-logs-swift` won't work ebcause there's no
"openstack" module.

I don't know how to make zuul-executor error persistent, but let's at
least make the life easier for the next poor sod installing this SW on
a system where "virtualenv" is "too old" -- which is also the latest
available Ubuntu if I'm checking their revision numbers correctly.
Require virtualenv >= 20 while preserving the blacklist from commit
7c6503aba. Given that Zuul's own CI uses PIP to pull the latest
virtualenv for the test suite, make sure that the users get what was
tested.

Change-Id: I2b53e532b63a9ae20e817d9599c46cccf613e425

Replace our tox-py37 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.

Change-Id: I4b382bb89847d3ae2b20292d923c3c2211dfb254

The global tox installation on our test nodes is affected by an
upstream issue[1].  Additionally, the virtualenv created by tox
under those conditions is also affected.  To work around this,
pin the version of importlib-resources in the global tox install,
as well as in Zuul's own requirements.

[1] https://gitlab.com/python-devs/importlib_resources/issues/83

Change-Id: I31ed50185a71d867a2ad512ef9b526c5b607ed5c

The canMerge check is executed whenever zuul tests if a change can
enter a gate pipeline. This is part of the critical path in the event
handling of the scheduler and therefore must be as fast as
possible. Currently this takes five requests for doing its work and
also transfers large amounts of data that is unneeded:

* get pull request
* get branch protection settings
* get commits
* get status of latest commit
* get check runs of latest commit

Especially when Github is busy this can slow down zuul's event
processing considerably. This can be optimized using graphql to only
query the data we need with a single request. This reduces requests
and load on Github and speeds up event processing in the scheduler.

Since this is the first usage of graphql this also sets up needed
testing infrastructure using graphene to mock the github api with real
test data.

Change-Id: I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10

GitPython was capped to <3.0.0 due to increase test times. According
to the upstream issue [1] this should, be fixed upstream. See if we
can uncap it now.

[1] https://github.com/gitpython-developers/GitPython/issues/906

Change-Id: I62a2731c4045f72060a67b8ad3eb89f6b07a0e97

There may have been a behavior change with polling that causes a
surplus of log lines.  Avoid that version until we understand more.

Change-Id: I246acda9352267eb6b9be6d1b06fb87b133709ec

virtualenv==20.0.0 introduced new symlinking of "seed" packages that
broke within a bwrap context. We fixed this by forcing those seed
packages to be installed by pip avoiding any symlinking. Since then
upstream virtualenv has realize this was bad default behavior and
virtualenv==20.0.2 does not symlink by default.

Take advantage of this and exclude 20.0.0 and 20.0.1 as they need
special handling. This allows other versions (new and old) to function.

Change-Id: I3b993e72b52e2234e046a657280b008887c535a3

New virtualenv has changed behavior and cli arguments. In particular the
seed packages are by default installed in a common lib location and not
updated. These packages are then symlinked to in the virtualenv. This
causes problems for zuul when we run ansible in bwrap and that common
lib path isn't bind mounted in the container.

We can fix this by telling virtualenv to install the seed packages uses
pip and not its internal bundled app data. This is preferable for Zuul
because then we don't have to manage permissions and locations of a
separate seed package library location.

Note that we force virtualenv to be >=20.0.0 as old virtualenv does not
have this new seeder option.

Change-Id: Ib9b2e0b51e6acd744d1c23b4a27067f0202a679d

The new virtualenv release 20.0.0 breaks tests so cap it for now.

Change-Id: I40660f31c17ddae13e05a75552baed125235fa05

CherryPy uses cheroot underneath as http server. The release 8.1.0
introduced a regression which leads to occasional connection resets
while handling concurrent requests [1]. Cap it until there is a
release available that fixes this issue.

[1] https://github.com/cherrypy/cheroot/issues/263

Change-Id: I39716e9c203cc42ab320cb47e090fbc8f67320c9

Instead of preventing newer version which would create conflicts, we
just avoid the known broken versions.

Change-Id: I7e04bd6efddaab9847c01c48a2e6ef4be1a41e4e

This is broken with Gerrit due to
https://github.com/urllib3/urllib3/pull/1684

Change-Id: Ie2c817bb91463cecc64e3022e11330898b11062c

This looks like it might be extending our unit test runtime by
50% (from 30 mins to 45, thefore pushing the overall job from
45 to 60+ mins).

Change-Id: I333cc1dbe5b2223e7d6f86ff7cdb6d5f063bda39

Allow an operator to define authorization rules.
Allow an operator to add authorization rules to a tenant.
Add a rule parser and a rule registry.

The authZ engine is not plugged in yet.

Change-Id: I3a86c6c7d62ad2bce68a98dbd2fff18549b94fb9

Don't use system virtualenv. Install it from source/pypi.

Change-Id: Idea7291cb6bff9d23c87fd40b3bb45f32115b7ef

As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in
follow-ups.

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927

This updates the github driver to cache PRs by sha using cachetool's
LRUCache.

We make this change because we need to cache closed PRs so can't rely on
the action of closing a PR to remove the PR from the cache. Since we
don't have a good method of evicting entries we fall back to LRU with
a reasonable cache size (2k commits).

Change-Id: I5fb6c8b33f9eed221a8b84e537f02e7dccf2d2df

If the gearman server vanishes (e.g. due to a VM crash) some clients
like the merger may not notice that it is gone. They just wait forever
for data to be received on an inactive connection. In our case the VM
containing the zuul-scheduler crashed and after the restart of the
scheduler all mergers were waiting for data on the stale connection
which blocked a successful scheduler restart.  Using tcp keepalive we
can detect that situation and let broken inactive connections be
killed by the kernel.

Depends-On: I8589cd45450245a25539c051355b38d16ee9f4b9
Change-Id: I30049d59d873d64f3b69c5587c775827e3545854

18.0.1 has been released an fixes:

  https://github.com/cherrypy/cherrypy/pull/1736

Change-Id: Iffda4376e1393020d54ba6e25d0dda07616d72c6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

With the latest cherrypy version the tests run into the hard test
timeout. This is caused by a header conversion issue introduced in
cherrypy 18.0.0. There is a pull request [1] that fixes this
issue. Once this is merged and released we can uncap it again.

The newest reno release, 2.10.0, requires a feature of newer sphinx without
specifying that the minimum compatibility has changed. Uncap sphinx here
so at least the doc build is not broken, use Sphinx 1.6.1 since that is
the oldest sphinx version supporting what reno needs.

[1] https://github.com/cherrypy/cherrypy/pull/1736

Change-Id: Ic00e664152062d1fc1677e467b638ac2be99aef1
Co-Authored-By: Andreas Jaeger <jaegerandi@gmail.com>

Change-Id: I480385a8b0e85266fdd77d251d0b748f1be028b0

* Aiohttp (and related libraries) have a python support policy
  which is causing us problems.
* Cherrypy supports threads which integrates well with the rest
  of Zuul.

Change-Id: Ib611df06035890d3e87fc5ad92fdfc7ac441edce

Change-Id: Iaae6ca9cc52d2c63821bc4266aef437457e6fe92
Story: 2002104

Our custom command.py Ansible module is updated to match the
version from 2.5, plus our additions.

strip_internal_keys() is moved within Ansible yet again.

Change-Id: Iab951c11b23a24757cf5334b36bc8f7d12e19db0
Depends-On: https://review.openstack.org/567007

async-timeout 3.0.0 has released and only supports python 3.5.3 (which
is greater than the Xenial we test on).  Pin it for lower pythons
(this is similar to I460652a4468bfa76895a5c563612ff6119c0d483 which
pinned yarl -- in that case it was stated it was 1.1.1 which would be
the final release for python<3.5.3 support; however in this case it's
not clear if maybe there might be more 2.X releases, so hence the
less-than qualifier...)

Change-Id: Ifa2ddedbd4f431d8cd08059e31814b95dc74e368

This change adds a MQTT reporter to publish build results message.

Change-Id: I5a9937a7952beac5c77d83ab791d48ff000b447b

As described in [1], the latest releases of yarl (brought in via
aiohttp) require python 3.5.3.  Pin it for lower versions (which
happens to be 3.5.2 in xenial ... which we test on) to the last
release that works (1.1.1).

[1] https://github.com/aio-libs/yarl/issues/189

Change-Id: I460652a4468bfa76895a5c563612ff6119c0d483

The github status requirements matching and trigger filter are
currently plain text matching based. This currently limits sharing of
pipeline definitions between tenants as zuul reports the status as
'<tenant>/<pipeline>'. This currently makes it necessary to define
trigger filter for each tenant [1] and completely blocks pipeline
requirements.

A solution to this is regex matching which makes it possible to define
the filter once [2].

Further this enables an interesting further use case to trigger on any
successfull status [3]. This makes it easier to cooperate with other
CI systems or github apps which also set a status.

Directly use re2 as this will be used in the future for regex
matching.

[1] Trigger filter snippet
  trigger:
    github:
      - event: pull_request
        action: status
        status:
          - zuul:tenant1/check:success
          - zuul:tenant2/check:success
          - zuul:tenant3/check:success
          - zuul:tenant4/check:success

[2] Regex trigger filter snippet
  trigger:
    github:
      - event: pull_request
        action: status
        status:
          - zuul:.+/check:success

[3] Generic success filter snippet
  trigger:
    github:
      - event: pull_request
        action: status
        status:
          - .*:success

Change-Id: Id1b9d7334db78d0f13db33d47a80ffdb65f921df

We need at least GitHub3.py 1.1.0 to pull in a bugfix which fixes
handling large patches [1].

[1] https://github.com/sigmavirus24/github3.py/pull/817

Change-Id: I3e83cae456c4211cfe8b3547fab5ed652c06fe76

This updates both the dependency to ansible 2.4 and also ports in the
needed changes to the command module.

Version 2.4.0 definitely does not work for us because YAML
hosts file parsing is broken, but 2.4.1 and greater should
be fine.

Change-Id: I63f72b45ecb9533eac5ba9eb0eef426beec905e3

This release fixes the issues preventing our use of it.

Change-Id: Ife6534360cce313096596226fca123d67b2c5536

Update to stats >=3.0 to be in-line with nodepool

Change-Id: Ib84655378bdb7c7c3c66bf6187b462b3be2f908d

The latest change on github3.py broke us [1]. As a temporary quick fix
pin it to a revision prior to this change until we have a proper way
how to deal with this.

[1] Trace:
2018-03-12 12:48:36,299 ERROR zuul.Scheduler: Exception in management event:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/github3/models.py", line 47, in __init__
    self._update_attributes(json)
  File "/usr/lib/python3.6/site-packages/github3/repos/repo.py", line 2720, in _update_attributes
    self.original_license = repo['license']
KeyError: 'license'

Change-Id: I22f62ea6b6d621a8b817542e72fe3a762a79a491

We don't use it anymore. We use aiohttp.

Change-Id: Iad3aa9421e8a317941a5f252bda9817806caee93

Detecting an unknown tenant got tricky when we started returning a
message about tenants not being ready yet. In order to be able to return
a 404 for tenants we legitimately do not know anything about, keep an
UnparsedAbideConfig on the scheduler so that we can check it in case of
a miss. If we know about a tenant but don't have a config, we can return
the 'not ready' message. If we don't know about it at all, we can throw
the 404.

Also, remove the custom handler test. We have tests in other contexts
(like tests of the github webhook) that test the equivilent
functionality.

Change-Id: Icff5d7036b6a237646ad7482103f7b487621bac0

This is to work around the following issue:

  aiohttp requires Python '>=3.5.3' but the running Python is 3.5.2

Which is breaking builds on Ubuntu Xenial.

Change-Id: I656f4e37a12159b60154cd868a5416c6af3e3139
Signed-off-by: Paul Belanger <pabelanger@redhat.com>