Replace master with feature/zuulv3
Change-Id: I5d9e9a573e9bffd6e06c73b2412f12c92169d8a3
This commit is contained in:
James E. Blair
commit
3f3b744263
|
|
@ -3,14 +3,17 @@
|
|||
*.egg-info
|
||||
*.pyc
|
||||
.idea
|
||||
.mypy_cache
|
||||
.test
|
||||
.testrepository
|
||||
.tox
|
||||
.venv
|
||||
.coverage
|
||||
AUTHORS
|
||||
build/*
|
||||
ChangeLog
|
||||
config
|
||||
doc/build/*
|
||||
zuul/versioninfo
|
||||
dist/
|
||||
cover/
|
||||
htmlcov/
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
[DEFAULT]
|
||||
test_command=OS_LOG_LEVEL=${OS_LOG_LEVEL:-INFO} OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} OS_LOG_DEFAULTS=${OS_LOG_DEFAULTS:-""} ${PYTHON:-python} -m subunit.run discover -t ./ tests $LISTOPT $IDOPTION
|
||||
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} OS_LOG_DEFAULTS=${OS_LOG_DEFAULTS:-""} ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./tests/unit} $LISTOPT $IDOPTION
|
||||
test_id_option=--load-list $IDFILE
|
||||
test_list_option=--list
|
||||
|
|
|
|||
62
.zuul.yaml
62
.zuul.yaml
|
|
@ -1,14 +1,64 @@
|
|||
- nodeset:
|
||||
name: zuul-functional-temp-master
|
||||
nodes:
|
||||
- name: controller
|
||||
label: ubuntu-xenial
|
||||
- name: node1
|
||||
label: ubuntu-xenial
|
||||
- name: node2
|
||||
label: ubuntu-xenial
|
||||
groups:
|
||||
- name: node
|
||||
nodes:
|
||||
- node1
|
||||
- node2
|
||||
|
||||
- job:
|
||||
name: zuul-stream-functional
|
||||
parent: multinode
|
||||
nodeset: zuul-functional-temp-master
|
||||
pre-run: playbooks/zuul-stream/pre.yaml
|
||||
run: playbooks/zuul-stream/functional.yaml
|
||||
post-run:
|
||||
- playbooks/zuul-stream/post.yaml
|
||||
- playbooks/zuul-stream/post-ara.yaml
|
||||
required-projects:
|
||||
- openstack/ara
|
||||
files:
|
||||
- zuul/ansible/callback/.*
|
||||
- playbooks/zuul-stream/.*
|
||||
|
||||
- project:
|
||||
name: openstack-infra/zuul
|
||||
check:
|
||||
jobs:
|
||||
- tox-docs
|
||||
- tox-cover:
|
||||
voting: false
|
||||
- build-openstack-sphinx-docs:
|
||||
irrelevant-files:
|
||||
- zuul/cmd/migrate.py
|
||||
- playbooks/zuul-migrate/.*
|
||||
vars:
|
||||
sphinx_python: python3
|
||||
- tox-pep8
|
||||
- tox-py27
|
||||
- tox-py35:
|
||||
irrelevant-files:
|
||||
- zuul/cmd/migrate.py
|
||||
- playbooks/zuul-migrate/.*
|
||||
- zuul-stream-functional
|
||||
gate:
|
||||
jobs:
|
||||
- tox-docs
|
||||
- build-openstack-sphinx-docs:
|
||||
irrelevant-files:
|
||||
- zuul/cmd/migrate.py
|
||||
- playbooks/zuul-migrate/.*
|
||||
vars:
|
||||
sphinx_python: python3
|
||||
- tox-pep8
|
||||
- tox-py27
|
||||
- tox-py35:
|
||||
irrelevant-files:
|
||||
- zuul/cmd/migrate.py
|
||||
- playbooks/zuul-migrate/.*
|
||||
- zuul-stream-functional
|
||||
post:
|
||||
jobs:
|
||||
- publish-openstack-sphinx-docs-infra-python3
|
||||
- publish-openstack-python-branch-tarball
|
||||
|
|
|
|||
5
NEWS.rst
5
NEWS.rst
|
|
@ -12,11 +12,6 @@ Since 2.0.0:
|
|||
the Zuul server in smaller deployments. Several configuration
|
||||
options have moved from the ``zuul`` section to ``merger``.
|
||||
|
||||
* Gerrit label names must now be listed in your layout.yaml exactly as
|
||||
they appear in Gerrit. This means case and special characters must
|
||||
match. This change was made to accomodate Gerrit 2.13 which needs the
|
||||
strings to match for changes to be successfully submitted.
|
||||
|
||||
Since 1.3.0:
|
||||
|
||||
* The Jenkins launcher is replaced with Gearman launcher. An internal
|
||||
|
|
|
|||
143
README.rst
143
README.rst
|
|
@ -3,6 +3,21 @@ Zuul
|
|||
|
||||
Zuul is a project gating system developed for the OpenStack Project.
|
||||
|
||||
We are currently engaged in a significant development effort in
|
||||
preparation for the third major version of Zuul. We call this effort
|
||||
`Zuul v3`_ and it is described in more detail below.
|
||||
|
||||
The latest documentation for Zuul v3 is published at:
|
||||
https://docs.openstack.org/infra/zuul/feature/zuulv3/
|
||||
|
||||
If you are looking for the Edge routing service named Zuul that is
|
||||
related to Netflix, it can be found here:
|
||||
https://github.com/Netflix/zuul
|
||||
|
||||
If you are looking for the Javascript testing tool named Zuul, it
|
||||
can be found here:
|
||||
https://github.com/defunctzombie/zuul
|
||||
|
||||
Contributing
|
||||
------------
|
||||
|
||||
|
|
@ -25,3 +40,131 @@ that links to your launchpad account). Example::
|
|||
# Do your commits
|
||||
$ git review
|
||||
# Enter your username if prompted
|
||||
|
||||
Zuul v3
|
||||
-------
|
||||
|
||||
The Zuul v3 effort involves significant changes to Zuul, and its
|
||||
companion program, Nodepool. The intent is for Zuul to become more
|
||||
generally useful outside of the OpenStack community. This is the best
|
||||
way to get started with this effort:
|
||||
|
||||
1) Read the Zuul v3 spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
|
||||
|
||||
We use specification documents like this to describe large efforts
|
||||
where we want to make sure that all the participants are in
|
||||
agreement about what will happen and generally how before starting
|
||||
development. These specs should contain enough information for
|
||||
people to evaluate the proposal generally, and sometimes include
|
||||
specific details that need to be agreed upon in advance. They are
|
||||
living documents which can change as work gets underway. However,
|
||||
every change or detail does not need to be reflected in the spec --
|
||||
most work is simply done with patches (and revised if necessary in
|
||||
code review).
|
||||
|
||||
2) Read the Nodepool build-workers spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html
|
||||
|
||||
3) Review any proposed updates to these specs: https://review.openstack.org/#/q/status:open+project:openstack-infra/infra-specs+topic:zuulv3
|
||||
|
||||
Some of the information in the specs may be effectively superceded
|
||||
by changes here, which are still undergoing review.
|
||||
|
||||
4) Read developer documentation on the internal data model and testing: http://docs.openstack.org/infra/zuul/feature/zuulv3/developer.html
|
||||
|
||||
The general philosophy for Zuul tests is to perform functional
|
||||
testing of either the individual component or the entire end-to-end
|
||||
system with external systems (such as Gerrit) replaced with fakes.
|
||||
Before adding additional unit tests with a narrower focus, consider
|
||||
whether they add value to this system or are merely duplicative of
|
||||
functional tests.
|
||||
|
||||
5) Review open changes: https://review.openstack.org/#/q/status:open+branch:feature/zuulv3
|
||||
|
||||
We find that the most valuable code reviews are ones that spot
|
||||
problems with the proposed change, or raise questions about how
|
||||
that might affect other systems or subsequent work. It is also a
|
||||
great way to stay involved as a team in work performed by others
|
||||
(for instance, by observing and asking questions about development
|
||||
while it is in progress). We try not to sweat the small things and
|
||||
don't worry too much about style suggestions or other nitpicky
|
||||
things (unless they are relevant -- for instance, a -1 vote on a
|
||||
change that introduces a yaml change out of character with existing
|
||||
conventions is useful because it makes the system more
|
||||
user-friendly; a -1 vote on a change which uses a sub-optimal line
|
||||
breaking strategy is probably not the best use of anyone's time).
|
||||
|
||||
6) Join #zuul on Freenode. Let others (especially jeblair who is
|
||||
trying to coordinate and prioritize work) know what you would like
|
||||
to work on.
|
||||
|
||||
7) Check storyboard for status of current work items: https://storyboard.openstack.org/#!/board/41
|
||||
|
||||
Work items tagged with ``low-hanging-fruit`` are tasks that have
|
||||
been identified as not requiring an expansive knowledge of the
|
||||
system. They may still require either some knowledge or
|
||||
investigation into a specific area, but should be suitable for a
|
||||
developer who is becoming acquainted with the system. Those items
|
||||
can be found at:
|
||||
https://storyboard.openstack.org/#!/story/list?tags=low-hanging-fruit&tags=zuulv3
|
||||
|
||||
Once you are up to speed on those items, it will be helpful to know
|
||||
the following:
|
||||
|
||||
* Zuul v3 includes some substantial changes to Zuul, and in order to
|
||||
implement them quickly and simultaneously, we temporarily disabled
|
||||
most of the test suite. That test suite still has relevance, but
|
||||
tests are likely to need updating individually, with reasons ranging
|
||||
from something simple such as a test-framework method changing its
|
||||
name, to more substantial issues, such as a feature being removed as
|
||||
part of the v3 work. Each test will need to be evaluated
|
||||
individually. Feel free to, at any time, claim a test name in this
|
||||
story and work on re-enabling it:
|
||||
https://storyboard.openstack.org/#!/story/2000773
|
||||
|
||||
* Because of the importance of external systems, as well as the number
|
||||
of internal Zuul components, actually running Zuul in a development
|
||||
mode quickly becomes unweildy (imagine uploading changes to Gerrit
|
||||
repeatedly while altering Zuul source code). Instead, the best way
|
||||
to develop with Zuul is in fact to write a functional test.
|
||||
Construct a test to fully simulate the series of events you want to
|
||||
see, then run it in the foreground. For example::
|
||||
|
||||
.tox/py27/bin/python -m testtools.run tests.unit.test_scheduler.TestScheduler.test_jobs_executed
|
||||
|
||||
See TESTING.rst for more information.
|
||||
|
||||
* There are many occasions, when working on sweeping changes to Zuul
|
||||
v3, we left notes for future work items in the code marked with
|
||||
"TODOv3". These represent potentially serious missing functionality
|
||||
or other issues which must be resolved before an initial v3 release
|
||||
(unlike a more conventional TODO note, these really can not be left
|
||||
indefinitely). These present an opportunity to identify work items
|
||||
not otherwise tracked. The names associated with TODO or TODOv3
|
||||
items do not mean that only that person can address them -- they
|
||||
simply reflect who to ask to explain the item in more detail if it
|
||||
is too cryptic. In your own work, feel free to leave TODOv3 notes
|
||||
if a change would otherwise become too large or unweildy.
|
||||
|
||||
Python Version Support
|
||||
----------------------
|
||||
|
||||
Zuul v3 requires Python 3. It does not support Python 2.
|
||||
|
||||
As Ansible is used for the execution of jobs, it's important to note that
|
||||
while Ansible does support Python 3, not all of Ansible's modules do. Zuul
|
||||
currently sets ``ansible_python_interpreter`` to python2 so that remote
|
||||
content will be executed with Python2.
|
||||
|
||||
Roadmap
|
||||
-------
|
||||
|
||||
* Begin using Zuul v3 to run jobs for Zuul itself
|
||||
* Implement a shim to translate Zuul v2 demand into Nodepool Zookeeper
|
||||
launcher requests
|
||||
* Begin using Zookeeper based Nodepool launchers with Zuul v2.5 in
|
||||
OpenStack Infra
|
||||
* Move OpenStack Infra to use Zuul v3
|
||||
* Implement Github support
|
||||
* Begin using Zuul v3 to run tests on Ansible repos
|
||||
* Implement support in Nodepool for non-OpenStack clouds
|
||||
* Add native container support to Zuul / Nodepool
|
||||
|
|
|
|||
14
TESTING.rst
14
TESTING.rst
|
|
@ -17,6 +17,16 @@ More information on pip here: http://www.pip-installer.org/en/latest/
|
|||
|
||||
pip install tox
|
||||
|
||||
As of zuul v3, a running zookeeper is required to execute tests.
|
||||
|
||||
*Install zookeeper*::
|
||||
|
||||
[apt-get | yum] install zookeeperd
|
||||
|
||||
*Start zookeeper*::
|
||||
|
||||
service zookeeper start
|
||||
|
||||
Run The Tests
|
||||
-------------
|
||||
|
||||
|
|
@ -54,12 +64,12 @@ To run individual tests with tox::
|
|||
|
||||
For example, to *run the basic Zuul test*::
|
||||
|
||||
tox -e py27 -- tests.test_scheduler.TestScheduler.test_jobs_launched
|
||||
tox -e py27 -- tests.unit.test_scheduler.TestScheduler.test_jobs_executed
|
||||
|
||||
To *run one test in the foreground* (after previously having run tox
|
||||
to set up the virtualenv)::
|
||||
|
||||
.tox/py27/bin/python -m testtools.run tests.test_scheduler.TestScheduler.test_jobs_launched
|
||||
.tox/py27/bin/python -m testtools.run tests.unit.test_scheduler.TestScheduler.test_jobs_executed
|
||||
|
||||
List Failing Tests
|
||||
------------------
|
||||
|
|
|
|||
13
bindep.txt
13
bindep.txt
|
|
@ -4,3 +4,16 @@
|
|||
mysql-client [test]
|
||||
mysql-server [test]
|
||||
libjpeg-dev [test]
|
||||
openssl [test]
|
||||
zookeeperd [platform:dpkg]
|
||||
build-essential [platform:dpkg]
|
||||
gcc [platform:rpm]
|
||||
graphviz [doc]
|
||||
libssl-dev [platform:dpkg]
|
||||
openssl-devel [platform:rpm]
|
||||
libffi-dev [platform:dpkg]
|
||||
libffi-devel [platform:rpm]
|
||||
python-dev [platform:dpkg]
|
||||
python-devel [platform:rpm]
|
||||
bubblewrap [platform:rpm]
|
||||
redhat-rpm-config [platform:rpm]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,115 @@
|
|||
:title: Zuul Client
|
||||
|
||||
Zuul Client
|
||||
===========
|
||||
|
||||
Zuul includes a simple command line client that may be used by
|
||||
administrators to affect Zuul's behavior while running. It must be
|
||||
run on a host that has access to the Gearman server (e.g., locally on
|
||||
the Zuul host).
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The client uses the same zuul.conf file as the server, and will look
|
||||
for it in the same locations if not specified on the command line.
|
||||
|
||||
Usage
|
||||
-----
|
||||
The general options that apply to all subcommands are:
|
||||
|
||||
.. program-output:: zuul --help
|
||||
|
||||
The following subcommands are supported:
|
||||
|
||||
Autohold
|
||||
^^^^^^^^
|
||||
.. program-output:: zuul autohold --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul autohold --tenant openstack --project example_project --job example_job --reason "reason text" --count 1
|
||||
|
||||
Enqueue
|
||||
^^^^^^^
|
||||
.. program-output:: zuul enqueue --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul enqueue --tenant openstack --trigger gerrit --pipeline check --project example_project --change 12345,1
|
||||
|
||||
Note that the format of change id is <number>,<patchset>.
|
||||
|
||||
Enqueue-ref
|
||||
^^^^^^^^^^^
|
||||
|
||||
.. program-output:: zuul enqueue-ref --help
|
||||
|
||||
This command is provided to manually simulate a trigger from an
|
||||
external source. It can be useful for testing or replaying a trigger
|
||||
that is difficult or impossible to recreate at the source. The
|
||||
arguments to ``enqueue-ref`` will vary depending on the source and
|
||||
type of trigger. Some familiarity with the arguments emitted by
|
||||
``gerrit`` `update hooks
|
||||
<https://gerrit-review.googlesource.com/admin/projects/plugins/hooks>`__
|
||||
such as ``patchset-created`` and ``ref-updated`` is recommended. Some
|
||||
examples of common operations are provided below.
|
||||
|
||||
Manual enqueue examples
|
||||
***********************
|
||||
|
||||
It is common to have a ``release`` pipeline that listens for new tags
|
||||
coming from ``gerrit`` and performs a range of code packaging jobs.
|
||||
If there is an unexpected issue in the release jobs, the same tag can
|
||||
not be recreated in ``gerrit`` and the user must either tag a new
|
||||
release or request a manual re-triggering of the jobs. To re-trigger
|
||||
the jobs, pass the failed tag as the ``ref`` argument and set
|
||||
``newrev`` to the change associated with the tag in the project
|
||||
repository (i.e. what you see from ``git show X.Y.Z``)::
|
||||
|
||||
zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/example_project --ref refs/tags/X.Y.Z --newrev abc123...
|
||||
|
||||
The command can also be used asynchronosly trigger a job in a
|
||||
``periodic`` pipeline that would usually be run at a specific time by
|
||||
the ``timer`` driver. For example, the following command would
|
||||
trigger the ``periodic`` jobs against the current ``master`` branch
|
||||
top-of-tree for a project::
|
||||
|
||||
zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master
|
||||
|
||||
Another common pipeline is a ``post`` queue listening for ``gerrit``
|
||||
merge results. Triggering here is slightly more complicated as you
|
||||
wish to recreate the full ``ref-updated`` event from ``gerrit``. For
|
||||
a new commit on ``master``, the gerrit ``ref-updated`` trigger
|
||||
expresses "reset ``refs/heads/master`` for the project from ``oldrev``
|
||||
to ``newrev``" (``newrev`` being the committed change). Thus to
|
||||
replay the event, you could ``git log`` in the project and take the
|
||||
current ``HEAD`` and the prior change, then enqueue the event::
|
||||
|
||||
NEW_REF=$(git rev-parse HEAD)
|
||||
OLD_REF=$(git rev-parse HEAD~1)
|
||||
|
||||
zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline post --project openstack/example_project --ref refs/heads/master --newrev $NEW_REF --oldrev $OLD_REF
|
||||
|
||||
Note that zero values for ``oldrev`` and ``newrev`` can indicate
|
||||
branch creation and deletion; the source code is the best reference
|
||||
for these more advanced operations.
|
||||
|
||||
|
||||
Promote
|
||||
^^^^^^^
|
||||
.. program-output:: zuul promote --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul promote --tenant openstack --pipeline check --changes 12345,1 13336,3
|
||||
|
||||
Note that the format of changes id is <number>,<patchset>.
|
||||
|
||||
Show
|
||||
^^^^
|
||||
.. program-output:: zuul show --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul show running-jobs
|
||||
|
|
@ -0,0 +1,739 @@
|
|||
:title: Components
|
||||
|
||||
.. _components:
|
||||
|
||||
Components
|
||||
==========
|
||||
|
||||
Zuul is a distributed system consisting of several components, each of
|
||||
which is described below.
|
||||
|
||||
.. graphviz::
|
||||
:align: center
|
||||
|
||||
graph {
|
||||
node [shape=box]
|
||||
Gearman [shape=ellipse]
|
||||
Gerrit [fontcolor=grey]
|
||||
Zookeeper [shape=ellipse]
|
||||
Nodepool
|
||||
GitHub [fontcolor=grey]
|
||||
|
||||
Merger -- Gearman
|
||||
Executor -- Gearman
|
||||
Web -- Gearman
|
||||
|
||||
Gearman -- Scheduler;
|
||||
Scheduler -- Gerrit;
|
||||
Scheduler -- Zookeeper;
|
||||
Zookeeper -- Nodepool;
|
||||
Scheduler -- GitHub;
|
||||
}
|
||||
|
||||
Each of the Zuul processes may run on the same host, or different
|
||||
hosts. Within Zuul, the components communicate with the scheduler via
|
||||
the Gearman protocol, so each Zuul component needs to be able to
|
||||
connect to the host running the Gearman server (the scheduler has a
|
||||
built-in Gearman server which is recommended) on the Gearman port --
|
||||
TCP port 4730 by default.
|
||||
|
||||
The Zuul scheduler communicates with Nodepool via the ZooKeeper
|
||||
protocol. Nodepool requires an external ZooKeeper cluster, and the
|
||||
Zuul scheduler needs to be able to connect to the hosts in that
|
||||
cluster on TCP port 2181.
|
||||
|
||||
Both the Nodepool launchers and Zuul executors need to be able to
|
||||
communicate with the hosts which nodepool provides. If these are on
|
||||
private networks, the Executors will need to be able to route traffic
|
||||
to them.
|
||||
|
||||
If statsd is enabled, every service needs to be able to emit data to
|
||||
statsd. Statsd can be configured to run on each host and forward
|
||||
data, or services may emit to a centralized statsd collector. Statsd
|
||||
listens on UDP port 8125 by default.
|
||||
|
||||
All Zuul processes read the ``/etc/zuul/zuul.conf`` file (an alternate
|
||||
location may be supplied on the command line) which uses an INI file
|
||||
syntax. Each component may have its own configuration file, though
|
||||
you may find it simpler to use the same file for all components.
|
||||
|
||||
An example ``zuul.conf``:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[gearman]
|
||||
server=localhost
|
||||
|
||||
[gearman_server]
|
||||
start=true
|
||||
log_config=/etc/zuul/gearman-logging.yaml
|
||||
|
||||
[zookeeper]
|
||||
hosts=zk1.example.com,zk2.example.com,zk3.example.com
|
||||
|
||||
[webapp]
|
||||
status_url=https://zuul.example.com/status
|
||||
|
||||
[scheduler]
|
||||
log_config=/etc/zuul/scheduler-logging.yaml
|
||||
|
||||
A minimal Zuul system may consist of a :ref:`scheduler` and
|
||||
:ref:`executor` both running on the same host. Larger installations
|
||||
should consider running multiple executors, each on a dedicated host,
|
||||
and running mergers on dedicated hosts as well.
|
||||
|
||||
Common
|
||||
------
|
||||
|
||||
The following applies to all Zuul components.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The following sections of ``zuul.conf`` are used by all Zuul components:
|
||||
|
||||
|
||||
.. attr:: gearman
|
||||
|
||||
Client connection information for Gearman.
|
||||
|
||||
.. attr:: server
|
||||
:required:
|
||||
|
||||
Hostname or IP address of the Gearman server.
|
||||
|
||||
.. attr:: port
|
||||
:default: 4730
|
||||
|
||||
Port on which the Gearman server is listening.
|
||||
|
||||
.. attr:: ssl_ca
|
||||
|
||||
An openssl file containing a set of concatenated “certification
|
||||
authority” certificates in PEM formet.
|
||||
|
||||
.. attr:: ssl_cert
|
||||
|
||||
An openssl file containing the client public certificate in PEM format.
|
||||
|
||||
.. attr:: ssl_key
|
||||
|
||||
An openssl file containing the client private key in PEM format.
|
||||
|
||||
.. attr:: statsd
|
||||
|
||||
Information about the optional statsd server. If the ``statsd``
|
||||
python module is installed and this section is configured,
|
||||
statistics will be reported to statsd. See :ref:`statsd` for more
|
||||
information.
|
||||
|
||||
.. attr:: server
|
||||
|
||||
Hostname or IP address of the statsd server.
|
||||
|
||||
.. attr:: port
|
||||
:default: 8125
|
||||
|
||||
The UDP port on which the statsd server is listening.
|
||||
|
||||
.. attr:: prefix
|
||||
|
||||
If present, this will be prefixed to all of the keys before
|
||||
transmitting to the statsd server.
|
||||
|
||||
.. NOTE: this is a white lie at this point, since only the scheduler
|
||||
uses this, however, we expect other components to use it later, so
|
||||
it's reasonable for admins to plan for this now.
|
||||
|
||||
.. attr:: zookeeper
|
||||
|
||||
Client connection information for ZooKeeper
|
||||
|
||||
.. attr:: hosts
|
||||
:required:
|
||||
|
||||
A list of zookeeper hosts for Zuul to use when communicating
|
||||
with Nodepool.
|
||||
|
||||
.. attr:: session_timeout
|
||||
:default: 10.0
|
||||
|
||||
The ZooKeeper session timeout, in seconds.
|
||||
|
||||
|
||||
.. _scheduler:
|
||||
|
||||
Scheduler
|
||||
---------
|
||||
|
||||
The scheduler is the primary component of Zuul. The scheduler is not
|
||||
a scalable component; one, and only one, scheduler must be running at
|
||||
all times for Zuul to be operational. It receives events from any
|
||||
connections to remote systems which have been configured, enqueues
|
||||
items into pipelines, distributes jobs to executors, and reports
|
||||
results.
|
||||
|
||||
The scheduler includes a Gearman server which is used to communicate
|
||||
with other components of Zuul. It is possible to use an external
|
||||
Gearman server, but the built-in server is well-tested and
|
||||
recommended. If the built-in server is used, other Zuul hosts will
|
||||
need to be able to connect to the scheduler on the Gearman port, TCP
|
||||
port 4730. It is also strongly recommended to use SSL certs with
|
||||
Gearman, as secrets are transferred from the scheduler to executors
|
||||
over this link.
|
||||
|
||||
The scheduler must be able to connect to the ZooKeeper cluster used by
|
||||
Nodepool in order to request nodes. It does not need to connect
|
||||
directly to the nodes themselves, however -- that function is handled
|
||||
by the Executors.
|
||||
|
||||
It must also be able to connect to any services for which connections
|
||||
are configured (Gerrit, GitHub, etc).
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The following sections of ``zuul.conf`` are used by the scheduler:
|
||||
|
||||
|
||||
.. attr:: gearman_server
|
||||
|
||||
The builtin gearman server. Zuul can fork a gearman process from
|
||||
itself rather than connecting to an external one.
|
||||
|
||||
.. attr:: start
|
||||
:default: false
|
||||
|
||||
Whether to start the internal Gearman server.
|
||||
|
||||
.. attr:: listen_address
|
||||
:default: all addresses
|
||||
|
||||
IP address or domain name on which to listen.
|
||||
|
||||
.. attr:: port
|
||||
:default: 4730
|
||||
|
||||
TCP port on which to listen.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file for internal Gearman server.
|
||||
|
||||
.. attr:: ssl_ca
|
||||
|
||||
An openssl file containing a set of concatenated “certification
|
||||
authority” certificates in PEM formet.
|
||||
|
||||
.. attr:: ssl_cert
|
||||
|
||||
An openssl file containing the server public certificate in PEM
|
||||
format.
|
||||
|
||||
.. attr:: ssl_key
|
||||
|
||||
An openssl file containing the server private key in PEM format.
|
||||
|
||||
.. attr:: webapp
|
||||
|
||||
.. attr:: listen_address
|
||||
:default: all addresses
|
||||
|
||||
IP address or domain name on which to listen.
|
||||
|
||||
.. attr:: port
|
||||
:default: 8001
|
||||
|
||||
Port on which the webapp is listening.
|
||||
|
||||
.. attr:: status_expiry
|
||||
:default: 1
|
||||
|
||||
Zuul will cache the status.json file for this many seconds.
|
||||
|
||||
.. attr:: status_url
|
||||
|
||||
URL that will be posted in Zuul comments made to changes when
|
||||
starting jobs for a change.
|
||||
|
||||
.. TODO: is this effectively required?
|
||||
|
||||
.. attr:: scheduler
|
||||
|
||||
.. attr:: command_socket
|
||||
:default: /var/lib/zuul/scheduler.socket
|
||||
|
||||
Path to command socket file for the scheduler process.
|
||||
|
||||
.. attr:: tenant_config
|
||||
:required:
|
||||
|
||||
Path to :ref:`tenant-config` file.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file.
|
||||
|
||||
.. attr:: pidfile
|
||||
:default: /var/run/zuul-schedurecr/zuul-scheduler.pid
|
||||
|
||||
Path to PID lock file.
|
||||
|
||||
.. attr:: state_dir
|
||||
:default: /var/lib/zuul
|
||||
|
||||
Path to directory in which Zuul should save its state.
|
||||
|
||||
Operation
|
||||
~~~~~~~~~
|
||||
|
||||
To start the scheduler, run ``zuul-scheduler``. To stop it, kill the
|
||||
PID which was saved in the pidfile specified in the configuration.
|
||||
|
||||
Most of Zuul's configuration is automatically updated as changes to
|
||||
the repositories which contain it are merged. However, Zuul must be
|
||||
explicitly notified of changes to the tenant config file, since it is
|
||||
not read from a git repository. To do so, send the scheduler PID
|
||||
(saved in the pidfile specified in the configuration) a `SIGHUP`
|
||||
signal.
|
||||
|
||||
Merger
|
||||
------
|
||||
|
||||
Mergers are an optional Zuul service; they are not required for Zuul
|
||||
to operate, but some high volume sites may benefit from running them.
|
||||
Zuul performs quite a lot of git operations in the course of its work.
|
||||
Each change that is to be tested must be speculatively merged with the
|
||||
current state of its target branch to ensure that it can merge, and to
|
||||
ensure that the tests that Zuul perform accurately represent the
|
||||
outcome of merging the change. Because Zuul's configuration is stored
|
||||
in the git repos it interacts with, and is dynamically evaluated, Zuul
|
||||
often needs to perform a speculative merge in order to determine
|
||||
whether it needs to perform any further actions.
|
||||
|
||||
All of these git operations add up, and while Zuul executors can also
|
||||
perform them, large numbers may impact their ability to run jobs.
|
||||
Therefore, administrators may wish to run standalone mergers in order
|
||||
to reduce the load on executors.
|
||||
|
||||
Mergers need to be able to connect to the Gearman server (usually the
|
||||
scheduler host) as well as any services for which connections are
|
||||
configured (Gerrit, GitHub, etc).
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The following section of ``zuul.conf`` is used by the merger:
|
||||
|
||||
.. attr:: merger
|
||||
|
||||
.. attr:: command_socket
|
||||
:default: /var/lib/zuul/merger.socket
|
||||
|
||||
Path to command socket file for the merger process.
|
||||
|
||||
.. attr:: git_dir
|
||||
|
||||
Directory in which Zuul should clone git repositories.
|
||||
|
||||
.. attr:: git_http_low_speed_limit
|
||||
:default: 1000
|
||||
|
||||
If the HTTP transfer speed is less then git_http_low_speed_limit for
|
||||
longer then git_http_low_speed_time, the transfer is aborted.
|
||||
|
||||
Value in bytes, setting to 0 will disable.
|
||||
|
||||
.. attr:: git_http_low_speed_time
|
||||
:default: 30
|
||||
|
||||
If the HTTP transfer speed is less then git_http_low_speed_limit for
|
||||
longer then git_http_low_speed_time, the transfer is aborted.
|
||||
|
||||
Value in seconds, setting to 0 will disable.
|
||||
|
||||
.. attr:: git_user_email
|
||||
|
||||
Value to pass to `git config user.email
|
||||
<https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
|
||||
|
||||
.. attr:: git_user_name
|
||||
|
||||
Value to pass to `git config user.name
|
||||
<https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file for the merger process.
|
||||
|
||||
.. attr:: pidfile
|
||||
:default: /var/run/zuul-merger/zuul-merger.pid
|
||||
|
||||
Path to PID lock file for the merger process.
|
||||
|
||||
Operation
|
||||
~~~~~~~~~
|
||||
|
||||
To start the merger, run ``zuul-merger``. To stop it, kill the
|
||||
PID which was saved in the pidfile specified in the configuration.
|
||||
|
||||
.. _executor:
|
||||
|
||||
Executor
|
||||
--------
|
||||
|
||||
Executors are responsible for running jobs. At the start of each job,
|
||||
an executor prepares an environment in which to run Ansible which
|
||||
contains all of the git repositories specified by the job with all
|
||||
dependent changes merged into their appropriate branches. The branch
|
||||
corresponding to the proposed change will be checked out (in all
|
||||
projects, if it exists). Any roles specified by the job will also be
|
||||
present (also with dependent changes merged, if appropriate) and added
|
||||
to the Ansible role path. The executor also prepares an Ansible
|
||||
inventory file with all of the nodes requested by the job.
|
||||
|
||||
The executor also contains a merger. This is used by the executor to
|
||||
prepare the git repositories used by jobs, but is also available to
|
||||
perform any tasks normally performed by standalone mergers. Because
|
||||
the executor performs both roles, small Zuul installations may not
|
||||
need to run standalone mergers.
|
||||
|
||||
Executors need to be able to connect to the Gearman server (usually
|
||||
the scheduler host), any services for which connections are configured
|
||||
(Gerrit, GitHub, etc), as well as directly to the hosts which Nodepool
|
||||
provides.
|
||||
|
||||
Trusted and Untrusted Playbooks
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The executor runs playbooks in one of two execution contexts depending
|
||||
on whether the project containing the playbook is a
|
||||
:term:`config-project` or an :term:`untrusted-project`. If the
|
||||
playbook is in a config project, the executor runs the playbook in the
|
||||
*trusted* execution context, otherwise, it is run in the *untrusted*
|
||||
execution context.
|
||||
|
||||
Both execution contexts use `bubblewrap`_ [#nullwrap]_ to create a
|
||||
namespace to ensure that playbook executions are isolated and are unable
|
||||
to access files outside of a restricted environment. The administrator
|
||||
may configure additional local directories on the executor to be made
|
||||
available to the restricted environment.
|
||||
|
||||
The trusted execution context has access to all Ansible features,
|
||||
including the ability to load custom Ansible modules. Needless to
|
||||
say, extra scrutiny should be given to code that runs in a trusted
|
||||
context as it could be used to compromise other jobs running on the
|
||||
executor, or the executor itself, especially if the administrator has
|
||||
granted additional access through bubblewrap, or a method of escaping
|
||||
the restricted environment created by bubblewrap is found.
|
||||
|
||||
Playbooks run in the untrusted execution context are not permitted to
|
||||
load additional Ansible modules or access files outside of the
|
||||
restricted environment prepared for them by the executor. In addition
|
||||
to the bubblewrap environment applied to both execution contexts, in
|
||||
the untrusted context some standard Ansible modules are replaced with
|
||||
versions which prohibit some actions, including attempts to access
|
||||
files outside of the restricted execution context. These redundant
|
||||
protections are made as part of a defense-in-depth strategy.
|
||||
|
||||
.. _bubblewrap: https://github.com/projectatomic/bubblewrap
|
||||
.. [#nullwrap] Unless one has set execution_wrapper to nullwrap in the
|
||||
executor configuration.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The following sections of ``zuul.conf`` are used by the executor:
|
||||
|
||||
.. attr:: executor
|
||||
|
||||
.. attr:: command_socket
|
||||
:default: /var/lib/zuul/executor.socket
|
||||
|
||||
Path to command socket file for the executor process.
|
||||
|
||||
.. attr:: finger_port
|
||||
:default: 7900
|
||||
|
||||
Port to use for finger log streamer.
|
||||
|
||||
.. attr:: git_dir
|
||||
:default: /var/lib/zuul/git
|
||||
|
||||
Directory that Zuul should clone local git repositories to. The
|
||||
executor keeps a local copy of every git repository it works
|
||||
with to speed operations and perform speculative merging.
|
||||
|
||||
This should be on the same filesystem as
|
||||
:attr:`executor.job_dir` so that when git repos are cloned into
|
||||
the job workspaces, they can be hard-linked to the local git
|
||||
cache.
|
||||
|
||||
.. attr:: job_dir
|
||||
:default: /tmp
|
||||
|
||||
Directory that Zuul should use to hold temporary job directories.
|
||||
When each job is run, a new entry will be created under this
|
||||
directory to hold the configuration and scratch workspace for
|
||||
that job. It will be deleted at the end of the job (unless the
|
||||
`--keep-jobdir` command line option is specified).
|
||||
|
||||
This should be on the same filesystem as :attr:`executor.git_dir`
|
||||
so that when git repos are cloned into the job workspaces, they
|
||||
can be hard-linked to the local git cache.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file for the executor process.
|
||||
|
||||
.. attr:: pidfile
|
||||
:default: /var/run/zuul-executor/zuul-executor.pid
|
||||
|
||||
Path to PID lock file for the executor process.
|
||||
|
||||
.. attr:: private_key_file
|
||||
:default: ~/.ssh/id_rsa
|
||||
|
||||
SSH private key file to be used when logging into worker nodes.
|
||||
|
||||
.. _admin_sitewide_variables:
|
||||
|
||||
.. attr:: variables
|
||||
|
||||
Path to an Ansible variables file to supply site-wide variables.
|
||||
This should be a YAML-formatted file consisting of a single
|
||||
dictionary. The contents will be made available to all jobs as
|
||||
Ansible variables. These variables take precedence over all
|
||||
other forms (job variables and secrets). Care should be taken
|
||||
when naming these variables to avoid potential collisions with
|
||||
those used by jobs. Prefixing variable names with a
|
||||
site-specific identifier is recommended. The default is not to
|
||||
add any site-wide variables. See the :ref:`User's Guide
|
||||
<user_sitewide_variables>` for more information.
|
||||
|
||||
.. attr:: disk_limit_per_job
|
||||
:default: 250
|
||||
|
||||
This integer is the maximum number of megabytes that any one job
|
||||
is allowed to consume on disk while it is running. If a job's
|
||||
scratch space has more than this much space consumed, it will be
|
||||
aborted.
|
||||
|
||||
.. attr:: trusted_ro_paths
|
||||
|
||||
List of paths, separated by ``:`` to read-only bind mount into
|
||||
trusted bubblewrap contexts.
|
||||
|
||||
.. attr:: trusted_rw_paths
|
||||
|
||||
List of paths, separated by ``:`` to read-write bind mount into
|
||||
trusted bubblewrap contexts.
|
||||
|
||||
.. attr:: untrusted_ro_paths
|
||||
|
||||
List of paths, separated by ``:`` to read-only bind mount into
|
||||
untrusted bubblewrap contexts.
|
||||
|
||||
.. attr:: untrusted_rw_paths
|
||||
|
||||
List of paths, separated by ``:`` to read-write bind mount into
|
||||
untrusted bubblewrap contexts.
|
||||
|
||||
.. attr:: execution_wrapper
|
||||
:default: bubblewrap
|
||||
|
||||
Name of the execution wrapper to use when executing
|
||||
`ansible-playbook`. The default, `bubblewrap` is recommended for
|
||||
all installations.
|
||||
|
||||
There is also a `nullwrap` driver for situations where one wants
|
||||
to run Zuul without access to bubblewrap or in such a way that
|
||||
bubblewrap may interfere with the jobs themselves. However,
|
||||
`nullwrap` is considered unsafe, as `bubblewrap` provides
|
||||
significant protections against malicious users and accidental
|
||||
breakage in playbooks. As such, `nullwrap` is not recommended
|
||||
for use in production.
|
||||
|
||||
This option, and thus, `nullwrap`, may be removed in the future.
|
||||
`bubblewrap` has become integral to securely operating Zuul. If you
|
||||
have a valid use case for it, we encourage you to let us know.
|
||||
|
||||
.. attr:: load_multiplier
|
||||
:default: 2.5
|
||||
|
||||
When an executor host gets too busy, the system may suffer
|
||||
timeouts and other ill effects. The executor will stop accepting
|
||||
more than 1 job at a time until load has lowered below a safe
|
||||
level. This level is determined by multiplying the number of
|
||||
CPU's by `load_multiplier`.
|
||||
|
||||
So for example, if the system has 2 CPUs, and load_multiplier
|
||||
is 2.5, the safe load for the system is 5.00. Any time the
|
||||
system load average is over 5.00, the executor will quit
|
||||
accepting multiple jobs at one time.
|
||||
|
||||
The executor will observe system load and determine whether
|
||||
to accept more jobs every 30 seconds.
|
||||
|
||||
.. attr:: hostname
|
||||
:default: hostname of the server
|
||||
|
||||
The executor needs to know its hostname under which it is reachable by
|
||||
zuul-web. Otherwise live console log streaming doesn't work. In most cases
|
||||
This is automatically detected correctly. But when running in environments
|
||||
where it cannot determine its hostname correctly this can be overridden
|
||||
here.
|
||||
|
||||
.. attr:: merger
|
||||
|
||||
.. attr:: git_user_email
|
||||
|
||||
Value to pass to `git config user.email
|
||||
<https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
|
||||
|
||||
.. attr:: git_user_name
|
||||
|
||||
Value to pass to `git config user.name
|
||||
<https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
|
||||
|
||||
Operation
|
||||
~~~~~~~~~
|
||||
|
||||
To start the executor, run ``zuul-executor``.
|
||||
|
||||
There are several commands which can be run to control the executor's
|
||||
behavior once it is running.
|
||||
|
||||
To stop the executor immediately, aborting all jobs (they may be
|
||||
relaunched according to their retry policy), run ``zuul-executor
|
||||
stop``.
|
||||
|
||||
To request that the executor stop executing new jobs and exit when all
|
||||
currently running jobs have completed, run ``zuul-executor graceful``.
|
||||
|
||||
To enable or disable running Ansible in verbose mode (with the
|
||||
``-vvv`` argument to ansible-playbook) run ``zuul-executor verbose``
|
||||
and ``zuul-executor unverbose``.
|
||||
|
||||
Web Server
|
||||
----------
|
||||
|
||||
The Zuul web server currently acts as a websocket interface to live log
|
||||
streaming. Eventually, it will serve as the single process handling all
|
||||
HTTP interactions with Zuul.
|
||||
|
||||
Web servers need to be able to connect to the Gearman server (usually
|
||||
the scheduler host). If the SQL reporter is used, they need to be
|
||||
able to connect to the database it reports to in order to support the
|
||||
dashboard. If a GitHub connection is configured, they need to be
|
||||
reachable by GitHub so they may receive notifications.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
In addition to the common configuration sections, the following
|
||||
sections of ``zuul.conf`` are used by the web server:
|
||||
|
||||
.. attr:: web
|
||||
|
||||
.. attr:: listen_address
|
||||
:default: 127.0.0.1
|
||||
|
||||
IP address or domain name on which to listen.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file for the web server process.
|
||||
|
||||
.. attr:: pidfile
|
||||
:default: /var/run/zuul-web/zuul-web.pid
|
||||
|
||||
Path to PID lock file for the web server process.
|
||||
|
||||
.. attr:: port
|
||||
:default: 9000
|
||||
|
||||
Port to use for web server process.
|
||||
|
||||
.. attr:: websocket_url
|
||||
|
||||
Base URL on which the websocket service is exposed, if different
|
||||
than the base URL of the web app.
|
||||
|
||||
.. attr:: static_cache_expiry
|
||||
:default: 3600
|
||||
|
||||
The Cache-Control max-age response header value for static files served
|
||||
by the zuul-web. Set to 0 during development to disable Cache-Control.
|
||||
|
||||
Operation
|
||||
~~~~~~~~~
|
||||
|
||||
To start the web server, run ``zuul-web``. To stop it, kill the
|
||||
PID which was saved in the pidfile specified in the configuration.
|
||||
|
||||
Finger Gateway
|
||||
--------------
|
||||
|
||||
The Zuul finger gateway listens on the standard finger port (79) for
|
||||
finger requests specifying a build UUID for which it should stream log
|
||||
results. The gateway will determine which executor is currently running that
|
||||
build and query that executor for the log stream.
|
||||
|
||||
This is intended to be used with the standard finger command line client.
|
||||
For example::
|
||||
|
||||
finger UUID@zuul.example.com
|
||||
|
||||
The above would stream the logs for the build identified by `UUID`.
|
||||
|
||||
Finger gateway servers need to be able to connect to the Gearman
|
||||
server (usually the scheduler host), as well as the console streaming
|
||||
port on the executors (usually 7900).
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
In addition to the common configuration sections, the following
|
||||
sections of ``zuul.conf`` are used by the finger gateway:
|
||||
|
||||
.. attr:: fingergw
|
||||
|
||||
.. attr:: command_socket
|
||||
:default: /var/lib/zuul/fingergw.socket
|
||||
|
||||
Path to command socket file for the executor process.
|
||||
|
||||
.. attr:: listen_address
|
||||
:default: all addresses
|
||||
|
||||
IP address or domain name on which to listen.
|
||||
|
||||
.. attr:: log_config
|
||||
|
||||
Path to log config file for the finger gateway process.
|
||||
|
||||
.. attr:: pidfile
|
||||
:default: /var/run/zuul-fingergw/zuul-fingergw.pid
|
||||
|
||||
Path to PID lock file for the finger gateway process.
|
||||
|
||||
.. attr:: port
|
||||
:default: 79
|
||||
|
||||
Port to use for the finger gateway. Note that since command line
|
||||
finger clients cannot usually specify the port, leaving this set to
|
||||
the default value is highly recommended.
|
||||
|
||||
.. attr:: user
|
||||
:default: zuul
|
||||
|
||||
User ID for the zuul-fingergw process. In normal operation as a
|
||||
daemon, the finger gateway should be started as the ``root`` user, but
|
||||
it will drop privileges to this user during startup.
|
||||
|
||||
Operation
|
||||
~~~~~~~~~
|
||||
|
||||
To start the finger gateway, run ``zuul-fingergw``. To stop it, kill the
|
||||
PID which was saved in the pidfile specified in the configuration.
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
:title: Connection Configuration
|
||||
|
||||
.. _connection-config:
|
||||
|
||||
Connection Configuration
|
||||
========================
|
||||
|
||||
Most of Zuul's configuration is contained in the git repositories upon
|
||||
which Zuul operates, however, some configuration outside of git
|
||||
repositories is still required to bootstrap the system. This includes
|
||||
information on connections between Zuul and other systems, as well as
|
||||
identifying the projects Zuul uses.
|
||||
|
||||
.. _connections:
|
||||
|
||||
Connections
|
||||
-----------
|
||||
|
||||
In order to interact with external systems, Zuul must have a
|
||||
*connection* to that system configured. Zuul includes a number of
|
||||
drivers, each of which implements the functionality necessary to
|
||||
connect to a system. Each connection in Zuul is associated with a
|
||||
driver.
|
||||
|
||||
To configure a connection in Zuul, select a unique name for the
|
||||
connection and add a section to ``zuul.conf`` with the form
|
||||
``[connection NAME]``. For example, a connection to a gerrit server
|
||||
may appear as:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[connection mygerritserver]
|
||||
driver=gerrit
|
||||
server=review.example.com
|
||||
|
||||
Zuul needs to use a single connection to look up information about
|
||||
changes hosted by a given system. When it looks up changes, it will
|
||||
do so using the first connection it finds that matches the server name
|
||||
it's looking for. It's generally best to use only a single connection
|
||||
for a given server, however, if you need more than one (for example,
|
||||
to satisfy unique reporting requirements) be sure to list the primary
|
||||
connection first as that is what Zuul will use to look up all changes
|
||||
for that server.
|
||||
|
||||
.. _drivers:
|
||||
|
||||
Drivers
|
||||
-------
|
||||
|
||||
Drivers may support any of the following functions:
|
||||
|
||||
* Sources -- hosts git repositories for projects. Zuul can clone git
|
||||
repos for projects and fetch refs.
|
||||
* Triggers -- emits events to which Zuul may respond. Triggers are
|
||||
configured in pipelines to cause changes or other refs to be
|
||||
enqueued.
|
||||
* Reporters -- outputs information when a pipeline is finished
|
||||
processing an item.
|
||||
|
||||
Zuul includes the following drivers:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
drivers/gerrit
|
||||
drivers/github
|
||||
drivers/git
|
||||
drivers/smtp
|
||||
drivers/sql
|
||||
drivers/timer
|
||||
drivers/zuul
|
||||
|
|
@ -0,0 +1,299 @@
|
|||
:title: Gerrit Driver
|
||||
|
||||
Gerrit
|
||||
======
|
||||
|
||||
`Gerrit`_ is a code review system. The Gerrit driver supports
|
||||
sources, triggers, and reporters.
|
||||
|
||||
.. _Gerrit: https://www.gerritcodereview.com/
|
||||
|
||||
Zuul will need access to a Gerrit user.
|
||||
|
||||
Create an SSH keypair for Zuul to use if there isn't one already, and
|
||||
create a Gerrit user with that key::
|
||||
|
||||
cat ~/id_rsa.pub | ssh -p29418 review.example.com gerrit create-account --ssh-key - --full-name Zuul zuul
|
||||
|
||||
Give that user whatever permissions will be needed on the projects you
|
||||
want Zuul to report on. For instance, you may want to grant
|
||||
``Verified +/-1`` and ``Submit`` to the user. Additional categories
|
||||
or values may be added to Gerrit. Zuul is very flexible and can take
|
||||
advantage of those.
|
||||
|
||||
Connection Configuration
|
||||
------------------------
|
||||
|
||||
The supported options in ``zuul.conf`` connections are:
|
||||
|
||||
.. attr:: <gerrit connection>
|
||||
|
||||
.. attr:: driver
|
||||
:required:
|
||||
|
||||
.. value:: gerrit
|
||||
|
||||
The connection must set ``driver=gerrit`` for Gerrit connections.
|
||||
|
||||
.. attr:: server
|
||||
|
||||
Fully qualified domain name of Gerrit server.
|
||||
|
||||
.. attr:: canonical_hostname
|
||||
|
||||
The canonical hostname associated with the git repos on the
|
||||
Gerrit server. Defaults to the value of
|
||||
:attr:`<gerrit connection>.server`. This is used to identify
|
||||
projects from this connection by name and in preparing repos on
|
||||
the filesystem for use by jobs. Note that Zuul will still only
|
||||
communicate with the Gerrit server identified by ``server``;
|
||||
this option is useful if users customarily use a different
|
||||
hostname to clone or pull git repos so that when Zuul places
|
||||
them in the job's working directory, they appear under this
|
||||
directory name.
|
||||
|
||||
.. attr:: port
|
||||
:default: 29418
|
||||
|
||||
Gerrit server port.
|
||||
|
||||
.. attr:: baseurl
|
||||
|
||||
Path to Gerrit web interface.
|
||||
|
||||
.. attr:: gitweb_url_template
|
||||
:default: {baseurl}/gitweb?p={project.name}.git;a=commitdiff;h={sha}
|
||||
|
||||
Url template for links to specific git shas. By default this will
|
||||
point at Gerrit's built in gitweb but you can customize this value
|
||||
to point elsewhere (like cgit or github).
|
||||
|
||||
The three values available for string interpolation are baseurl
|
||||
which points back to Gerrit, project and all of its safe attributes,
|
||||
and sha which is the git sha1.
|
||||
|
||||
.. attr:: user
|
||||
:default: zuul
|
||||
|
||||
User name to use when logging into Gerrit via ssh.
|
||||
|
||||
.. attr:: sshkey
|
||||
:default: ~zuul/.ssh/id_rsa
|
||||
|
||||
Path to SSH key to use when logging into Gerrit.
|
||||
|
||||
.. attr:: keepalive
|
||||
:default: 60
|
||||
|
||||
SSH connection keepalive timeout; ``0`` disables.
|
||||
|
||||
Trigger Configuration
|
||||
---------------------
|
||||
|
||||
Zuul works with standard versions of Gerrit by invoking the ``gerrit
|
||||
stream-events`` command over an SSH connection. It also reports back
|
||||
to Gerrit using SSH.
|
||||
|
||||
If using Gerrit 2.7 or later, make sure the user is a member of a group
|
||||
that is granted the ``Stream Events`` permission, otherwise it will not
|
||||
be able to invoke the ``gerrit stream-events`` command over SSH.
|
||||
|
||||
.. attr:: pipeline.trigger.<gerrit source>
|
||||
|
||||
The dictionary passed to the Gerrit pipeline ``trigger`` attribute
|
||||
supports the following attributes:
|
||||
|
||||
.. attr:: event
|
||||
:required:
|
||||
|
||||
The event name from gerrit. Examples: ``patchset-created``,
|
||||
``comment-added``, ``ref-updated``. This field is treated as a
|
||||
regular expression.
|
||||
|
||||
.. attr:: branch
|
||||
|
||||
The branch associated with the event. Example: ``master``.
|
||||
This field is treated as a regular expression, and multiple
|
||||
branches may be listed.
|
||||
|
||||
.. attr:: ref
|
||||
|
||||
On ref-updated events, the branch parameter is not used, instead
|
||||
the ref is provided. Currently Gerrit has the somewhat
|
||||
idiosyncratic behavior of specifying bare refs for branch names
|
||||
(e.g., ``master``), but full ref names for other kinds of refs
|
||||
(e.g., ``refs/tags/foo``). Zuul matches this value exactly
|
||||
against what Gerrit provides. This field is treated as a
|
||||
regular expression, and multiple refs may be listed.
|
||||
|
||||
.. attr:: ignore-deletes
|
||||
:default: true
|
||||
|
||||
When a branch is deleted, a ref-updated event is emitted with a
|
||||
newrev of all zeros specified. The ``ignore-deletes`` field is a
|
||||
boolean value that describes whether or not these newrevs
|
||||
trigger ref-updated events.
|
||||
|
||||
.. attr:: approval
|
||||
|
||||
This is only used for ``comment-added`` events. It only matches
|
||||
if the event has a matching approval associated with it.
|
||||
Example: ``Code-Review: 2`` matches a ``+2`` vote on the code
|
||||
review category. Multiple approvals may be listed.
|
||||
|
||||
.. attr:: email
|
||||
|
||||
This is used for any event. It takes a regex applied on the
|
||||
performer email, i.e. Gerrit account email address. If you want
|
||||
to specify several email filters, you must use a YAML list.
|
||||
Make sure to use non greedy matchers and to escapes dots!
|
||||
Example: ``email: ^.*?@example\.org$``.
|
||||
|
||||
.. attr:: username
|
||||
|
||||
This is used for any event. It takes a regex applied on the
|
||||
performer username, i.e. Gerrit account name. If you want to
|
||||
specify several username filters, you must use a YAML list.
|
||||
Make sure to use non greedy matchers and to escapes dots.
|
||||
Example: ``username: ^zuul$``.
|
||||
|
||||
.. attr:: comment
|
||||
|
||||
This is only used for ``comment-added`` events. It accepts a
|
||||
list of regexes that are searched for in the comment string. If
|
||||
any of these regexes matches a portion of the comment string the
|
||||
trigger is matched. ``comment: retrigger`` will match when
|
||||
comments containing ``retrigger`` somewhere in the comment text
|
||||
are added to a change.
|
||||
|
||||
.. attr:: require-approval
|
||||
|
||||
This may be used for any event. It requires that a certain kind
|
||||
of approval be present for the current patchset of the change
|
||||
(the approval could be added by the event in question). It
|
||||
follows the same syntax as :attr:`pipeline.require.<gerrit
|
||||
source>.approval`. For each specified criteria there must exist
|
||||
a matching approval.
|
||||
|
||||
.. attr:: reject-approval
|
||||
|
||||
This takes a list of approvals in the same format as
|
||||
:attr:`pipeline.trigger.<gerrit source>.require-approval` but
|
||||
will fail to enter the pipeline if there is a matching approval.
|
||||
|
||||
Reporter Configuration
|
||||
----------------------
|
||||
|
||||
Zuul works with standard versions of Gerrit by invoking the
|
||||
``gerrit`` command over an SSH connection. It reports back to
|
||||
Gerrit using SSH.
|
||||
|
||||
The dictionary passed to the Gerrit reporter is used for ``gerrit
|
||||
review`` arguments, with the boolean value of ``true`` simply
|
||||
indicating that the argument should be present without following it
|
||||
with a value. For example, ``verified: 1`` becomes ``gerrit review
|
||||
--verified 1`` and ``submit: true`` becomes ``gerrit review
|
||||
--submit``.
|
||||
|
||||
A :ref:`connection<connections>` that uses the gerrit driver must be
|
||||
supplied to the trigger.
|
||||
|
||||
Requirements Configuration
|
||||
--------------------------
|
||||
|
||||
As described in :attr:`pipeline.require` and :attr:`pipeline.reject`,
|
||||
pipelines may specify that items meet certain conditions in order to
|
||||
be enqueued into the pipeline. These conditions vary according to the
|
||||
source of the project in question. To supply requirements for changes
|
||||
from a Gerrit source named ``my-gerrit``, create a configuration such
|
||||
as the following:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
pipeline:
|
||||
require:
|
||||
my-gerrit:
|
||||
approval:
|
||||
- Code-Review: 2
|
||||
|
||||
This indicates that changes originating from the Gerrit connection
|
||||
named ``my-gerrit`` must have a ``Code-Review`` vote of ``+2`` in
|
||||
order to be enqueued into the pipeline.
|
||||
|
||||
.. attr:: pipeline.require.<gerrit source>
|
||||
|
||||
The dictionary passed to the Gerrit pipeline `require` attribute
|
||||
supports the following attributes:
|
||||
|
||||
.. attr:: approval
|
||||
|
||||
This requires that a certain kind of approval be present for the
|
||||
current patchset of the change (the approval could be added by
|
||||
the event in question). It takes several sub-parameters, all of
|
||||
which are optional and are combined together so that there must
|
||||
be an approval matching all specified requirements.
|
||||
|
||||
.. attr:: username
|
||||
|
||||
If present, an approval from this username is required. It is
|
||||
treated as a regular expression.
|
||||
|
||||
.. attr:: email
|
||||
|
||||
If present, an approval with this email address is required. It is
|
||||
treated as a regular expression.
|
||||
|
||||
.. attr:: older-than
|
||||
|
||||
If present, the approval must be older than this amount of time
|
||||
to match. Provide a time interval as a number with a suffix of
|
||||
"w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s"
|
||||
(seconds). Example ``48h`` or ``2d``.
|
||||
|
||||
.. attr:: newer-than
|
||||
|
||||
If present, the approval must be newer than this amount
|
||||
of time to match. Same format as "older-than".
|
||||
|
||||
Any other field is interpreted as a review category and value
|
||||
pair. For example ``Verified: 1`` would require that the
|
||||
approval be for a +1 vote in the "Verified" column. The value
|
||||
may either be a single value or a list: ``Verified: [1, 2]``
|
||||
would match either a +1 or +2 vote.
|
||||
|
||||
.. attr:: open
|
||||
|
||||
A boolean value (``true`` or ``false``) that indicates whether
|
||||
the change must be open or closed in order to be enqueued.
|
||||
|
||||
.. attr:: current-patchset
|
||||
|
||||
A boolean value (``true`` or ``false``) that indicates whether the
|
||||
change must be the current patchset in order to be enqueued.
|
||||
|
||||
.. attr:: status
|
||||
|
||||
A string value that corresponds with the status of the change
|
||||
reported by the trigger.
|
||||
|
||||
.. attr:: pipeline.reject.<gerrit source>
|
||||
|
||||
The `reject` attribute is the mirror of the `require` attribute. It
|
||||
also accepts a dictionary under the connection name. This
|
||||
dictionary supports the following attributes:
|
||||
|
||||
.. attr:: approval
|
||||
|
||||
This takes a list of approvals. If an approval matches the
|
||||
provided criteria the change can not be entered into the
|
||||
pipeline. It follows the same syntax as
|
||||
:attr:`pipeline.require.<gerrit source>.approval`.
|
||||
|
||||
Example to reject a change with any negative vote:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
reject:
|
||||
my-gerrit:
|
||||
approval:
|
||||
- Code-Review: [-1, -2]
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
:title: Git Driver
|
||||
|
||||
Git
|
||||
===
|
||||
|
||||
This driver can be used to load Zuul configuration from public Git repositories,
|
||||
for instance from ``openstack-infra/zuul-jobs`` that is suitable for use by
|
||||
any Zuul system. It can also be used to trigger jobs from ``ref-updated`` events
|
||||
in a pipeline.
|
||||
|
||||
Connection Configuration
|
||||
------------------------
|
||||
|
||||
The supported options in ``zuul.conf`` connections are:
|
||||
|
||||
.. attr:: <git connection>
|
||||
|
||||
.. attr:: driver
|
||||
:required:
|
||||
|
||||
.. value:: git
|
||||
|
||||
The connection must set ``driver=git`` for Git connections.
|
||||
|
||||
.. attr:: baseurl
|
||||
|
||||
Path to the base Git URL. Git repos name will be appended to it.
|
||||
|
||||
.. attr:: poll_delay
|
||||
:default: 7200
|
||||
|
||||
The delay in seconds of the Git repositories polling loop.
|
||||
|
||||
Trigger Configuration
|
||||
---------------------
|
||||
|
||||
.. attr:: pipeline.trigger.<git source>
|
||||
|
||||
The dictionary passed to the Git pipeline ``trigger`` attribute
|
||||
supports the following attributes:
|
||||
|
||||
.. attr:: event
|
||||
:required:
|
||||
|
||||
Only ``ref-updated`` is supported.
|
||||
|
||||
.. attr:: ref
|
||||
|
||||
On ref-updated events, a ref such as ``refs/heads/master`` or
|
||||
``^refs/tags/.*$``. This field is treated as a regular expression,
|
||||
and multiple refs may be listed.
|
||||
|
||||
.. attr:: ignore-deletes
|
||||
:default: true
|
||||
|
||||
When a ref is deleted, a ref-updated event is emitted with a
|
||||
newrev of all zeros specified. The ``ignore-deletes`` field is a
|
||||
boolean value that describes whether or not these newrevs
|
||||
trigger ref-updated events.
|
||||
|
|
@ -0,0 +1,458 @@
|
|||
:title: GitHub Driver
|
||||
|
||||
GitHub
|
||||
======
|
||||
|
||||
The GitHub driver supports sources, triggers, and reporters. It can
|
||||
interact with the public GitHub service as well as site-local
|
||||
installations of GitHub enterprise.
|
||||
|
||||
Configure GitHub
|
||||
----------------
|
||||
|
||||
There are two options currently available. GitHub's project owner can either
|
||||
manually setup web-hook or install a GitHub Application. In the first case,
|
||||
the project's owner needs to know the zuul endpoint and the webhook secrets.
|
||||
|
||||
|
||||
Web-Hook
|
||||
........
|
||||
|
||||
To configure a project's `webhook events
|
||||
<https://developer.github.com/webhooks/creating/>`_:
|
||||
|
||||
* Set *Payload URL* to
|
||||
``http://<zuul-hostname>/connection/<connection-name>/payload``.
|
||||
|
||||
* Set *Content Type* to ``application/json``.
|
||||
|
||||
Select *Events* you are interested in. See below for the supported events.
|
||||
|
||||
You will also need to have a GitHub user created for your zuul:
|
||||
|
||||
* Zuul public key needs to be added to the GitHub account
|
||||
|
||||
* A api_token needs to be created too, see this `article
|
||||
<https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_
|
||||
|
||||
Then in the zuul.conf, set webhook_token and api_token.
|
||||
|
||||
Application
|
||||
...........
|
||||
|
||||
To create a `GitHub application
|
||||
<https://developer.github.com/apps/building-integrations/setting-up-and-registering-github-apps/registering-github-apps/>`_:
|
||||
|
||||
* Go to your organization settings page to create the application, e.g.:
|
||||
https://github.com/organizations/my-org/settings/apps/new
|
||||
|
||||
* Set GitHub App name to "my-org-zuul"
|
||||
|
||||
* Set Setup URL to your setup documentation, when user install the application
|
||||
they are redirected to this url
|
||||
|
||||
* Set Webhook URL to
|
||||
``http://<zuul-hostname>/connection/<connection-name>/payload``.
|
||||
|
||||
* Create a Webhook secret
|
||||
|
||||
* Set permissions:
|
||||
|
||||
* Commit statuses: Read & Write
|
||||
|
||||
* Issues: Read & Write
|
||||
|
||||
* Pull requests: Read & Write
|
||||
|
||||
* Repository contents: Read & Write (write to let zuul merge change)
|
||||
|
||||
* Set events subscription:
|
||||
|
||||
* Label
|
||||
|
||||
* Status
|
||||
|
||||
* Issue comment
|
||||
|
||||
* Issues
|
||||
|
||||
* Pull request
|
||||
|
||||
* Pull request review
|
||||
|
||||
* Pull request review comment
|
||||
|
||||
* Commit comment
|
||||
|
||||
* Create
|
||||
|
||||
* Push
|
||||
|
||||
* Release
|
||||
|
||||
* Set Where can this GitHub App be installed to "Any account"
|
||||
|
||||
* Create the App
|
||||
|
||||
* Generate a Private key in the app settings page
|
||||
|
||||
Then in the zuul.conf, set webhook_token, app_id and app_key.
|
||||
After restarting zuul-scheduler, verify in the 'Advanced' tab that the
|
||||
Ping payload works (green tick and 200 response)
|
||||
|
||||
Users can now install the application using its public page, e.g.:
|
||||
https://github.com/apps/my-org-zuul
|
||||
|
||||
|
||||
Connection Configuration
|
||||
------------------------
|
||||
|
||||
There are two forms of operation. Either the Zuul installation can be
|
||||
configured as a `Github App`_ or it can be configured as a Webhook.
|
||||
|
||||
If the `Github App`_ approach is taken, the config settings ``app_id`` and
|
||||
``app_key`` are required. If the Webhook approach is taken, the ``api_token``
|
||||
setting is required.
|
||||
|
||||
The supported options in ``zuul.conf`` connections are:
|
||||
|
||||
.. attr:: <github connection>
|
||||
|
||||
.. attr:: driver
|
||||
:required:
|
||||
|
||||
.. value:: github
|
||||
|
||||
The connection must set ``driver=github`` for GitHub connections.
|
||||
|
||||
.. attr:: app_id
|
||||
|
||||
App ID if you are using a *GitHub App*. Can be found under the
|
||||
**Public Link** on the right hand side labeled **ID**.
|
||||
|
||||
.. attr:: app_key
|
||||
|
||||
Path to a file containing the secret key Zuul will use to create
|
||||
tokens for the API interactions. In Github this is known as
|
||||
**Private key** and must be collected when generated.
|
||||
|
||||
.. attr:: api_token
|
||||
|
||||
API token for accessing GitHub if Zuul is configured with
|
||||
Webhooks. See `Creating an access token for command-line use
|
||||
<https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_.
|
||||
|
||||
.. attr:: webhook_token
|
||||
|
||||
Required token for validating the webhook event payloads. In
|
||||
the GitHub App Configuration page, this is called **Webhook
|
||||
secret**. See `Securing your webhooks
|
||||
<https://developer.github.com/webhooks/securing/>`_.
|
||||
|
||||
.. attr:: sshkey
|
||||
:default: ~/.ssh/id_rsa
|
||||
|
||||
Path to SSH key to use when cloning github repositories.
|
||||
|
||||
.. attr:: server
|
||||
:default: github.com
|
||||
|
||||
Hostname of the github install (such as a GitHub Enterprise).
|
||||
|
||||
.. attr:: canonical_hostname
|
||||
|
||||
The canonical hostname associated with the git repos on the
|
||||
GitHub server. Defaults to the value of :attr:`<github
|
||||
connection>.server`. This is used to identify projects from
|
||||
this connection by name and in preparing repos on the filesystem
|
||||
for use by jobs. Note that Zuul will still only communicate
|
||||
with the GitHub server identified by **server**; this option is
|
||||
useful if users customarily use a different hostname to clone or
|
||||
pull git repos so that when Zuul places them in the job's
|
||||
working directory, they appear under this directory name.
|
||||
|
||||
.. attr:: verify_ssl
|
||||
:default: true
|
||||
|
||||
Enable or disable ssl verification for GitHub Enterprise. This
|
||||
is useful for a connection to a test installation.
|
||||
|
||||
Trigger Configuration
|
||||
---------------------
|
||||
GitHub webhook events can be configured as triggers.
|
||||
|
||||
A connection name with the GitHub driver can take multiple events with
|
||||
the following options.
|
||||
|
||||
.. attr:: pipeline.trigger.<github source>
|
||||
|
||||
The dictionary passed to the GitHub pipeline ``trigger`` attribute
|
||||
supports the following attributes:
|
||||
|
||||
.. attr:: event
|
||||
:required:
|
||||
|
||||
The event from github. Supported events are:
|
||||
|
||||
.. value:: pull_request
|
||||
|
||||
.. value:: pull_request_review
|
||||
|
||||
.. value:: push
|
||||
|
||||
.. attr:: action
|
||||
|
||||
A :value:`pipeline.trigger.<github source>.event.pull_request`
|
||||
event will have associated action(s) to trigger from. The
|
||||
supported actions are:
|
||||
|
||||
.. value:: opened
|
||||
|
||||
Pull request opened.
|
||||
|
||||
.. value:: changed
|
||||
|
||||
Pull request synchronized.
|
||||
|
||||
.. value:: closed
|
||||
|
||||
Pull request closed.
|
||||
|
||||
.. value:: reopened
|
||||
|
||||
Pull request reopened.
|
||||
|
||||
.. value:: comment
|
||||
|
||||
Comment added to pull request.
|
||||
|
||||
.. value:: labeled
|
||||
|
||||
Label added to pull request.
|
||||
|
||||
.. value:: unlabeled
|
||||
|
||||
Label removed from pull request.
|
||||
|
||||
.. value:: status
|
||||
|
||||
Status set on commit.
|
||||
|
||||
A :value:`pipeline.trigger.<github
|
||||
source>.event.pull_request_review` event will have associated
|
||||
action(s) to trigger from. The supported actions are:
|
||||
|
||||
.. value:: submitted
|
||||
|
||||
Pull request review added.
|
||||
|
||||
.. value:: dismissed
|
||||
|
||||
Pull request review removed.
|
||||
|
||||
.. attr:: branch
|
||||
|
||||
The branch associated with the event. Example: ``master``. This
|
||||
field is treated as a regular expression, and multiple branches
|
||||
may be listed. Used for ``pull_request`` and
|
||||
``pull_request_review`` events.
|
||||
|
||||
.. attr:: comment
|
||||
|
||||
This is only used for ``pull_request`` ``comment`` actions. It
|
||||
accepts a list of regexes that are searched for in the comment
|
||||
string. If any of these regexes matches a portion of the comment
|
||||
string the trigger is matched. ``comment: retrigger`` will
|
||||
match when comments containing 'retrigger' somewhere in the
|
||||
comment text are added to a pull request.
|
||||
|
||||
.. attr:: label
|
||||
|
||||
This is only used for ``labeled`` and ``unlabeled``
|
||||
``pull_request`` actions. It accepts a list of strings each of
|
||||
which matches the label name in the event literally. ``label:
|
||||
recheck`` will match a ``labeled`` action when pull request is
|
||||
labeled with a ``recheck`` label. ``label: 'do not test'`` will
|
||||
match a ``unlabeled`` action when a label with name ``do not
|
||||
test`` is removed from the pull request.
|
||||
|
||||
.. attr:: state
|
||||
|
||||
This is only used for ``pull_request_review`` events. It
|
||||
accepts a list of strings each of which is matched to the review
|
||||
state, which can be one of ``approved``, ``comment``, or
|
||||
``request_changes``.
|
||||
|
||||
.. attr:: status
|
||||
|
||||
This is used for ``pull-request`` and ``status`` actions. It
|
||||
accepts a list of strings each of which matches the user setting
|
||||
the status, the status context, and the status itself in the
|
||||
format of ``user:context:status``. For example,
|
||||
``zuul_github_ci_bot:check_pipeline:success``.
|
||||
|
||||
.. attr:: ref
|
||||
|
||||
This is only used for ``push`` events. This field is treated as
|
||||
a regular expression and multiple refs may be listed. GitHub
|
||||
always sends full ref name, eg. ``refs/tags/bar`` and this
|
||||
string is matched against the regular expression.
|
||||
|
||||
Reporter Configuration
|
||||
----------------------
|
||||
Zuul reports back to GitHub via GitHub API. Available reports include a PR
|
||||
comment containing the build results, a commit status on start, success and
|
||||
failure, an issue label addition/removal on the PR, and a merge of the PR
|
||||
itself. Status name, description, and context is taken from the pipeline.
|
||||
|
||||
.. attr:: pipeline.<reporter>.<github source>
|
||||
|
||||
To report to GitHub, the dictionaries passed to any of the pipeline
|
||||
:ref:`reporter<reporters>` attributes support the following
|
||||
attributes:
|
||||
|
||||
.. attr:: status
|
||||
|
||||
String value (``pending``, ``success``, ``failure``) that the
|
||||
reporter should set as the commit status on github.
|
||||
|
||||
.. TODO support role markup in :default: so we can xref
|
||||
:attr:`webapp.status_url` below
|
||||
|
||||
.. attr:: status-url
|
||||
:default: webapp.status_url or the empty string
|
||||
|
||||
String value for a link url to set in the github
|
||||
status. Defaults to the zuul server status_url, or the empty
|
||||
string if that is unset.
|
||||
|
||||
.. attr:: comment
|
||||
:default: true
|
||||
|
||||
Boolean value that determines if the reporter should add a
|
||||
comment to the pipeline status to the github pull request. Only
|
||||
used for Pull Request based items.
|
||||
|
||||
.. attr:: merge
|
||||
:default: false
|
||||
|
||||
Boolean value that determines if the reporter should merge the
|
||||
pull reqeust. Only used for Pull Request based items.
|
||||
|
||||
.. attr:: label
|
||||
|
||||
List of strings each representing an exact label name which
|
||||
should be added to the pull request by reporter. Only used for
|
||||
Pull Request based items.
|
||||
|
||||
.. attr:: unlabel
|
||||
|
||||
List of strings each representing an exact label name which
|
||||
should be removed from the pull request by reporter. Only used
|
||||
for Pull Request based items.
|
||||
|
||||
.. _Github App: https://developer.github.com/apps/
|
||||
|
||||
Requirements Configuration
|
||||
--------------------------
|
||||
|
||||
As described in :attr:`pipeline.require` and :attr:`pipeline.reject`,
|
||||
pipelines may specify that items meet certain conditions in order to
|
||||
be enqueued into the pipeline. These conditions vary according to the
|
||||
source of the project in question. To supply requirements for changes
|
||||
from a GitHub source named ``my-github``, create a congfiguration such
|
||||
as the following::
|
||||
|
||||
pipeline:
|
||||
require:
|
||||
my-github:
|
||||
review:
|
||||
- type: approval
|
||||
|
||||
This indicates that changes originating from the GitHub connection
|
||||
named ``my-github`` must have an approved code review in order to be
|
||||
enqueued into the pipeline.
|
||||
|
||||
.. attr:: pipeline.require.<github source>
|
||||
|
||||
The dictionary passed to the GitHub pipeline `require` attribute
|
||||
supports the following attributes:
|
||||
|
||||
.. attr:: review
|
||||
|
||||
This requires that a certain kind of code review be present for
|
||||
the pull request (it could be added by the event in question).
|
||||
It takes several sub-parameters, all of which are optional and
|
||||
are combined together so that there must be a code review
|
||||
matching all specified requirements.
|
||||
|
||||
.. attr:: username
|
||||
|
||||
If present, a code review from this username is required. It
|
||||
is treated as a regular expression.
|
||||
|
||||
.. attr:: email
|
||||
|
||||
If present, a code review with this email address is
|
||||
required. It is treated as a regular expression.
|
||||
|
||||
.. attr:: older-than
|
||||
|
||||
If present, the code review must be older than this amount of
|
||||
time to match. Provide a time interval as a number with a
|
||||
suffix of "w" (weeks), "d" (days), "h" (hours), "m"
|
||||
(minutes), "s" (seconds). Example ``48h`` or ``2d``.
|
||||
|
||||
.. attr:: newer-than
|
||||
|
||||
If present, the code review must be newer than this amount of
|
||||
time to match. Same format as "older-than".
|
||||
|
||||
.. attr:: type
|
||||
|
||||
If present, the code review must match this type (or types).
|
||||
|
||||
.. TODO: what types are valid?
|
||||
|
||||
.. attr:: permission
|
||||
|
||||
If present, the author of the code review must have this
|
||||
permission (or permissions). The available values are
|
||||
``read``, ``write``, and ``admin``.
|
||||
|
||||
.. attr:: open
|
||||
|
||||
A boolean value (``true`` or ``false``) that indicates whether
|
||||
the change must be open or closed in order to be enqueued.
|
||||
|
||||
.. attr:: current-patchset
|
||||
|
||||
A boolean value (``true`` or ``false``) that indicates whether
|
||||
the item must be associated with the latest commit in the pull
|
||||
request in order to be enqueued.
|
||||
|
||||
.. TODO: this could probably be expanded upon -- under what
|
||||
circumstances might this happen with github
|
||||
|
||||
.. attr:: status
|
||||
|
||||
A string value that corresponds with the status of the pull
|
||||
request. The syntax is ``user:status:value``.
|
||||
|
||||
.. attr:: label
|
||||
|
||||
A string value indicating that the pull request must have the
|
||||
indicated label (or labels).
|
||||
|
||||
.. attr:: pipeline.reject.<github source>
|
||||
|
||||
The `reject` attribute is the mirror of the `require` attribute. It
|
||||
also accepts a dictionary under the connection name. This
|
||||
dictionary supports the following attributes:
|
||||
|
||||
.. attr:: review
|
||||
|
||||
This takes a list of code reviews. If a code review matches the
|
||||
provided criteria the pull request can not be entered into the
|
||||
pipeline. It follows the same syntax as
|
||||
:attr:`pipeline.require.<github source>.review`
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
:title: SMTP Driver
|
||||
|
||||
SMTP
|
||||
====
|
||||
|
||||
The SMTP driver supports reporters only. It is used to send email
|
||||
when items report.
|
||||
|
||||
Connection Configuration
|
||||
------------------------
|
||||
|
||||
.. attr:: <smtp connection>
|
||||
|
||||
.. attr:: driver
|
||||
:required:
|
||||
|
||||
.. value:: smtp
|
||||
|
||||
The connection must set ``driver=smtp`` for SMTP connections.
|
||||
|
||||
.. attr:: server
|
||||
:default: localhost
|
||||
|
||||
SMTP server hostname or address to use.
|
||||
|
||||
.. attr:: port
|
||||
:default: 25
|
||||
|
||||
SMTP server port.
|
||||
|
||||
.. attr:: default_from
|
||||
:default: zuul
|
||||
|
||||
Who the email should appear to be sent from when emailing the report.
|
||||
This can be overridden by individual pipelines.
|
||||
|
||||
.. attr:: default_to
|
||||
:default: zuul
|
||||
|
||||
Who the report should be emailed to by default.
|
||||
This can be overridden by individual pipelines.
|
||||
|
||||
Reporter Configuration
|
||||
----------------------
|
||||
|
||||
A simple email reporter is also available.
|
||||
|
||||
A :ref:`connection<connections>` that uses the smtp driver must be supplied to the
|
||||
reporter. The connection also may specify a default *To* or *From*
|
||||
address.
|
||||
|
||||
Each pipeline can overwrite the ``subject`` or the ``to`` or ``from`` address by
|
||||
providing alternatives as arguments to the reporter. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- pipeline:
|
||||
name: post-merge
|
||||
success:
|
||||
outgoing_smtp:
|
||||
to: you@example.com
|
||||
failure:
|
||||
internal_smtp:
|
||||
to: you@example.com
|
||||
from: alternative@example.com
|
||||
subject: Change {change} failed
|
||||
|
||||
.. attr:: pipeline.<reporter>.<smtp source>
|
||||
|
||||
To report via email, the dictionaries passed to any of the pipeline
|
||||
:ref:`reporter<reporters>` attributes support the following
|
||||
attributes:
|
||||
|
||||
.. attr:: to
|
||||
|
||||
The SMTP recipient address for the report. Multiple addresses
|
||||
may be specified as one value separated by commas.
|
||||
|
||||
.. attr:: from
|
||||
|
||||
The SMTP sender address for the report.
|
||||
|
||||
.. attr:: subject
|
||||
|
||||
The Subject of the report email.
|
||||
|
||||
.. TODO: document subject string formatting.
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
:title: SQL Driver
|
||||
|
||||
SQL
|
||||
===
|
||||
|
||||
The SQL driver supports reporters only. Only one connection per
|
||||
database is permitted.
|
||||
|
||||
Connection Configuration
|
||||
------------------------
|
||||
|
||||
The connection options for the SQL driver are:
|
||||
|
||||
.. attr:: <sql connection>
|
||||
|
||||
.. attr:: driver
|
||||
:required:
|
||||
|
||||
.. value:: sql
|
||||
|
||||
The connection must set ``driver=sql`` for SQL connections.
|
||||
|
||||
.. attr:: dburi
|
||||
:required:
|
||||
|
||||
Database connection information in the form of a URI understood
|
||||
by SQLAlchemy. See `The SQLAlchemy manual
|
||||
<http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#database-urls>`_
|
||||
for more information.
|
||||
|
||||
The driver will automatically set up the database creating and managing
|
||||
the necesssary tables. Therefore the provided user should have sufficient
|
||||
permissions to manage the database. For example:
|
||||
|
||||
.. code-block:: sql
|
||||
|
||||
GRANT ALL ON my_database TO 'my_user'@'%';
|
||||
|
||||
.. attr:: pool_recycle
|
||||
:default: 1
|
||||
|
||||
Tune the pool_recycle value. See `The SQLAlchemy manual on pooling
|
||||
<http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle>`_
|
||||
for more information.
|
||||
|
||||
.. attr:: table_prefix
|
||||
:default: ''
|
||||
|
||||
The string to prefix the table names. This makes it possible to run
|
||||
several zuul deployments against the same database. This can be useful
|
||||
if you rely on external databases which you don't have under control.
|
||||
The default is to have no prefix.
|
||||
|
||||
Reporter Configuration
|
||||
----------------------
|
||||
|
||||
This reporter is used to store results in a database.
|
||||
|
||||
A :ref:`connection<connections>` that uses the sql driver must be
|
||||
supplied to the reporter.
|
||||
|
||||
``zuul.conf`` contains the database connection and credentials. To
|
||||
store different reports in different databases you'll need to create a
|
||||
new connection per database.
|
||||
|
||||
The SQL reporter does nothing on :attr:`pipeline.start` or
|
||||
:attr:`pipeline.merge-failure`; it only acts on
|
||||
:attr:`pipeline.success` or :attr:`pipeline.failure` reporting stages.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- pipeline:
|
||||
name: post-merge
|
||||
success:
|
||||
mydb_conn:
|
||||
failure:
|
||||
mydb_conn:
|
||||
|
||||
.. attr:: pipeline.<reporter>.<sql source>
|
||||
|
||||
To report to a database, add a key with the connection name and an
|
||||
empty value to the desired pipeline :ref:`reporter<reporters>`
|
||||
attributes.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
:title: Timer Driver
|
||||
|
||||
Timer
|
||||
=====
|
||||
|
||||
The timer driver supports triggers only. It is used for configuring
|
||||
pipelines so that jobs run at scheduled times. No connection
|
||||
configuration is required.
|
||||
|
||||
Trgger Configuration
|
||||
--------------------
|
||||
|
||||
Timers don't require a special connection or driver. Instead they can
|
||||
simply be used by listing ``timer`` as the trigger.
|
||||
|
||||
This trigger will run based on a cron-style time specification. It
|
||||
will enqueue an event into its pipeline for every project defined in
|
||||
the configuration. Any job associated with the pipeline will run in
|
||||
response to that event.
|
||||
|
||||
.. attr:: pipeline.trigger.timer
|
||||
|
||||
The timer trigger supports the following attributes:
|
||||
|
||||
.. attr:: time
|
||||
:required:
|
||||
|
||||
The time specification in cron syntax. Only the 5 part syntax
|
||||
is supported, not the symbolic names. Example: ``0 0 * * *``
|
||||
runs at midnight. The first weekday is Monday.
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
:title: Zuul Driver
|
||||
|
||||
Zuul
|
||||
====
|
||||
|
||||
The Zuul driver supports triggers only. It is used for triggering
|
||||
pipelines based on internal Zuul events.
|
||||
|
||||
Trigger Configuration
|
||||
---------------------
|
||||
|
||||
Zuul events don't require a special connection or driver. Instead they
|
||||
can simply be used by listing ``zuul`` as the trigger.
|
||||
|
||||
.. attr:: pipeline.trigger.zuul
|
||||
|
||||
The Zuul trigger supports the following attributes:
|
||||
|
||||
.. attr:: event
|
||||
:required:
|
||||
|
||||
The event name. Currently supported events:
|
||||
|
||||
.. value:: project-change-merged
|
||||
|
||||
When Zuul merges a change to a project, it generates this
|
||||
event for every open change in the project.
|
||||
|
||||
.. warning::
|
||||
|
||||
Triggering on this event can cause poor performance when
|
||||
using the GitHub driver with a large number of
|
||||
installations.
|
||||
|
||||
.. value:: parent-change-enqueued
|
||||
|
||||
When Zuul enqueues a change into any pipeline, it generates
|
||||
this event for every child of that change.
|
||||
|
||||
.. attr:: pipeline
|
||||
|
||||
Only available for ``parent-change-enqueued`` events. This is
|
||||
the name of the pipeline in which the parent change was
|
||||
enqueued.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
Administrator's Guide
|
||||
=====================
|
||||
|
||||
This guide is intended for administrators of Zuul systems. It covers
|
||||
installation, operation, and the portion of Zuul configuration that
|
||||
occurs outside of the projects upon which Zuul operates. Advanced
|
||||
users may be interested in some of the concepts described here, as
|
||||
well as understanding what features the underlying configuration
|
||||
provides to in-project configuration.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
quick-start
|
||||
installation
|
||||
components
|
||||
connections
|
||||
tenants
|
||||
monitoring
|
||||
client
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
Installation
|
||||
============
|
||||
|
||||
Install Zuul
|
||||
------------
|
||||
|
||||
To install a Zuul release from PyPI, run::
|
||||
|
||||
pip install zuul
|
||||
|
||||
Or from a git checkout, run::
|
||||
|
||||
pip install .
|
||||
|
||||
That will also install Zuul's python dependencies. To minimize
|
||||
interaction with other python packages installed on a system, you may
|
||||
wish to install Zuul within a Python virtualenv.
|
||||
|
||||
Zuul has several system-level dependencies as well. You can find a
|
||||
list of operating system packages in `bindep.txt` in Zuul's source
|
||||
directory.
|
||||
|
||||
External Dependencies
|
||||
---------------------
|
||||
|
||||
Zuul interacts with several other systems described below.
|
||||
|
||||
Gearman
|
||||
~~~~~~~
|
||||
|
||||
Gearman is a job distribution system that Zuul uses to communicate
|
||||
with its distributed components. The Zuul scheduler distributes work
|
||||
to Zuul mergers and executors using Gearman. You may supply your own
|
||||
gearman server, but the Zuul scheduler includes a built-in server
|
||||
which is recommended. Ensure that all Zuul hosts can communicate with
|
||||
the gearman server.
|
||||
|
||||
Zuul distributes secrets to executors via gearman, so be sure to
|
||||
secure it with TLS and certificate authentication. Obtain (or
|
||||
generate) a certificate for both the server and the clients (they may
|
||||
use the same certificate or have individual certificates). They must
|
||||
be signed by a CA, but it can be your own CA.
|
||||
|
||||
Nodepool
|
||||
~~~~~~~~
|
||||
|
||||
In order to run all but the simplest jobs, Zuul uses a companion
|
||||
program, Nodepool, to supply the nodes (whether dynamic cloud
|
||||
instances or static hardware) used by jobs. Before starting Zuul,
|
||||
ensure you have Nodepool installed and any images you require built.
|
||||
Zuul only makes one requirement of these nodes: that it be able to log
|
||||
in given a username and ssh private key.
|
||||
|
||||
.. TODO: SpamapS any zookeeper config recommendations?
|
||||
|
||||
Nodepool uses Zookeeper to communicate internally among its
|
||||
components, and also to communicate with Zuul. You can run a simple
|
||||
single-node Zookeeper instance, or a multi-node cluster. Ensure that
|
||||
the host running the Zuul scheduler has access to the cluster.
|
||||
|
||||
Ansible
|
||||
~~~~~~~
|
||||
|
||||
Zuul uses Ansible to run jobs. Each version of Zuul is designed to
|
||||
work with a specific, contemporary version of Ansible. Zuul specifies
|
||||
that version of Ansible in its python package metadata, and normally
|
||||
the correct version will be installed automatically with Zuul.
|
||||
Because of the close integration of Zuul and Ansible, attempting to
|
||||
use other versions of Ansible with Zuul is not recommended.
|
||||
|
|
@ -0,0 +1,278 @@
|
|||
:title: Monitoring
|
||||
|
||||
Monitoring
|
||||
==========
|
||||
|
||||
.. _statsd:
|
||||
|
||||
Statsd reporting
|
||||
----------------
|
||||
|
||||
Zuul comes with support for the statsd protocol, when enabled and configured
|
||||
(see below), the Zuul scheduler will emit raw metrics to a statsd receiver
|
||||
which let you in turn generate nice graphics.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Statsd support uses the ``statsd`` python module. Note that support
|
||||
is optional and Zuul will start without the statsd python module
|
||||
present.
|
||||
|
||||
Configuration is in the :attr:`statsd` section of ``zuul.conf``.
|
||||
|
||||
Metrics
|
||||
~~~~~~~
|
||||
|
||||
These metrics are emitted by the Zuul :ref:`scheduler`:
|
||||
|
||||
.. stat:: zuul.event.<driver>.event.<type>
|
||||
:type: counter
|
||||
|
||||
Zuul will report counters for each type of event it receives from
|
||||
each of its configured drivers.
|
||||
|
||||
.. stat:: zuul.tenant.<tenant>.pipeline
|
||||
|
||||
Holds metrics specific to jobs. This hierarchy includes:
|
||||
|
||||
.. stat:: <pipeline name>
|
||||
|
||||
A set of metrics for each pipeline named as defined in the Zuul
|
||||
config.
|
||||
|
||||
.. stat:: all_jobs
|
||||
:type: counter
|
||||
|
||||
Number of jobs triggered by the pipeline.
|
||||
|
||||
.. stat:: current_changes
|
||||
:type: gauge
|
||||
|
||||
The number of items currently being processed by this
|
||||
pipeline.
|
||||
|
||||
.. stat:: project
|
||||
|
||||
This hierarchy holds more specific metrics for each project
|
||||
participating in the pipeline.
|
||||
|
||||
.. stat:: <canonical_hostname>
|
||||
|
||||
The canonical hostname for the triggering project.
|
||||
Embedded ``.`` characters will be translated to ``_``.
|
||||
|
||||
.. stat:: <project>
|
||||
|
||||
The name of the triggering project. Embedded ``/`` or
|
||||
``.`` characters will be translated to ``_``.
|
||||
|
||||
.. stat:: <branch>
|
||||
|
||||
The name of the triggering branch. Embedded ``/`` or
|
||||
``.`` characters will be translated to ``_``.
|
||||
|
||||
.. stat:: job
|
||||
|
||||
Subtree detailing per-project job statistics:
|
||||
|
||||
.. stat:: <jobname>
|
||||
|
||||
The triggered job name.
|
||||
|
||||
.. stat:: <result>
|
||||
:type: counter, timer
|
||||
|
||||
A counter for each type of result (e.g., ``SUCCESS`` or
|
||||
``FAILURE``, ``ERROR``, etc.) for the job. If the
|
||||
result is ``SUCCESS`` or ``FAILURE``, Zuul will
|
||||
additionally report the duration of the build as a
|
||||
timer.
|
||||
|
||||
.. stat:: current_changes
|
||||
:type: gauge
|
||||
|
||||
The number of items of this project currently being
|
||||
processed by this pipeline.
|
||||
|
||||
.. stat:: resident_time
|
||||
:type: timer
|
||||
|
||||
A timer metric reporting how long each item for this
|
||||
project has been in the pipeline.
|
||||
|
||||
.. stat:: total_changes
|
||||
:type: counter
|
||||
|
||||
The number of changes for this project processed by the
|
||||
pipeline since Zuul started.
|
||||
|
||||
.. stat:: resident_time
|
||||
:type: timer
|
||||
|
||||
A timer metric reporting how long each item has been in the
|
||||
pipeline.
|
||||
|
||||
.. stat:: total_changes
|
||||
:type: counter
|
||||
|
||||
The number of changes processed by the pipeline since Zuul
|
||||
started.
|
||||
|
||||
.. stat:: wait_time
|
||||
:type: timer
|
||||
|
||||
How long each item spent in the pipeline before its first job
|
||||
started.
|
||||
|
||||
.. stat:: zuul.executor.<executor>
|
||||
|
||||
Holds metrics emitted by individual executors. The ``<executor>``
|
||||
component of the key will be replaced with the hostname of the
|
||||
executor.
|
||||
|
||||
.. stat:: builds
|
||||
:type: counter
|
||||
|
||||
Incremented each time the executor starts a build.
|
||||
|
||||
.. stat:: running_builds
|
||||
:type: gauge
|
||||
|
||||
The number of builds currently running on this executor.
|
||||
|
||||
.. stat:: load_average
|
||||
:type: gauge
|
||||
|
||||
The one-minute load average of this executor, multiplied by 100.
|
||||
|
||||
.. stat:: zuul.nodepool
|
||||
|
||||
Holds metrics related to Zuul requests from Nodepool.
|
||||
|
||||
.. stat:: requested
|
||||
:type: counter
|
||||
|
||||
Incremented each time a node request is submitted to Nodepool.
|
||||
|
||||
.. stat:: label.<label>
|
||||
:type: counter
|
||||
|
||||
Incremented each time a request for a specific label is
|
||||
submitted to Nodepool.
|
||||
|
||||
.. stat:: size.<size>
|
||||
:type: counter
|
||||
|
||||
Incremented each time a request of a specific size is submitted
|
||||
to Nodepool. For example, a request for 3 nodes would use the
|
||||
key ``zuul.nodepool.requested.size.3``.
|
||||
|
||||
.. stat:: canceled
|
||||
:type: counter, timer
|
||||
|
||||
The counter is incremented each time a node request is canceled
|
||||
by Zuul. The timer records the elapsed time from request to
|
||||
cancelation.
|
||||
|
||||
.. stat:: label.<label>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific label.
|
||||
|
||||
.. stat:: size.<size>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific request size.
|
||||
|
||||
.. stat:: fulfilled
|
||||
:type: counter, timer
|
||||
|
||||
The counter is incremented each time a node request is fulfilled
|
||||
by Nodepool. The timer records the elapsed time from request to
|
||||
fulfillment.
|
||||
|
||||
.. stat:: label.<label>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific label.
|
||||
|
||||
.. stat:: size.<size>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific request size.
|
||||
|
||||
.. stat:: failed
|
||||
:type: counter, timer
|
||||
|
||||
The counter is incremented each time Nodepool fails to fulfill a
|
||||
node request. The timer records the elapsed time from request
|
||||
to failure.
|
||||
|
||||
.. stat:: label.<label>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific label.
|
||||
|
||||
.. stat:: size.<size>
|
||||
:type: counter, timer
|
||||
|
||||
The same, for a specific request size.
|
||||
|
||||
.. stat:: current_requests
|
||||
:type: gauge
|
||||
|
||||
The number of outstanding nodepool requests from Zuul.
|
||||
|
||||
.. stat:: zuul.mergers
|
||||
|
||||
Holds metrics related to Zuul mergers.
|
||||
|
||||
.. stat:: online
|
||||
:type: gauge
|
||||
|
||||
The number of Zuul merger processes online.
|
||||
|
||||
.. stat:: jobs_running
|
||||
:type: gauge
|
||||
|
||||
The number of merge jobs running.
|
||||
|
||||
.. stat:: jobs_queued
|
||||
:type: gauge
|
||||
|
||||
The number of merge jobs queued.
|
||||
|
||||
.. stat:: zuul.executors
|
||||
|
||||
Holds metrics related to Zuul executors.
|
||||
|
||||
.. stat:: online
|
||||
:type: gauge
|
||||
|
||||
The number of Zuul executor processes online.
|
||||
|
||||
.. stat:: accepting
|
||||
:type: gauge
|
||||
|
||||
The number of Zuul executor processes accepting new jobs.
|
||||
|
||||
.. stat:: jobs_running
|
||||
:type: gauge
|
||||
|
||||
The number of executor jobs running.
|
||||
|
||||
.. stat:: jobs_queued
|
||||
:type: gauge
|
||||
|
||||
The number of executor jobs queued.
|
||||
|
||||
|
||||
As an example, given a job named `myjob` in `mytenant` triggered by a
|
||||
change to `myproject` on the `master` branch in the `gate` pipeline
|
||||
which took 40 seconds to build, the Zuul scheduler will emit the
|
||||
following statsd events:
|
||||
|
||||
* ``zuul.tenant.mytenant.pipeline.gate.project.example_com.myproject.master.job.myjob.SUCCESS`` +1
|
||||
* ``zuul.tenant.mytenant.pipeline.gate.project.example_com.myproject.master.job.myjob.SUCCESS`` 40 seconds
|
||||
* ``zuul.tenant.mytenant.pipeline.gate.all_jobs`` +1
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
Quick Start Guide
|
||||
=================
|
||||
|
||||
This provides a very simple overview of Zuul. It is recommended to
|
||||
read the following sections for more details.
|
||||
|
||||
Install Zuul
|
||||
------------
|
||||
|
||||
You can get zuul from pypi via::
|
||||
|
||||
pip install zuul
|
||||
|
||||
Zuul Components
|
||||
---------------
|
||||
|
||||
Zuul provides the following components:
|
||||
|
||||
- **zuul-scheduler**: The main Zuul process. Handles receiving
|
||||
events, executing jobs, collecting results and posting reports.
|
||||
Coordinates the work of the other components.
|
||||
|
||||
- **zuul-merger**: Scale-out component that performs git merge
|
||||
operations. Zuul performs a large number of git operations in
|
||||
the course of its work. Adding merger processes can help speed
|
||||
Zuul's processing. This component is optional (zero or more of
|
||||
these can be run).
|
||||
|
||||
- **zuul-executor**: Scale-out component for executing jobs. At
|
||||
least one of these is required. Depending on system
|
||||
configuration, you can expect a single executor to handle up to
|
||||
about 100 simultaneous jobs. Can handle the functions of a
|
||||
merger if dedicated mergers are not provided. One or more of
|
||||
these must be run.
|
||||
|
||||
- **zuul-web**: A web server that currently provides websocket access to
|
||||
live-streaming of logs.
|
||||
|
||||
- **gearman**: optional builtin gearman daemon provided by zuul-scheduler
|
||||
|
||||
External components:
|
||||
|
||||
- **gearman**: A gearman daemon if the built-in daemon is not used.
|
||||
|
||||
- **zookeeper**: A zookeeper cluster (or single host) for
|
||||
communicating with Nodepool.
|
||||
|
||||
- **nodepool**: Provides nodes for Zuul to use when executing jobs.
|
||||
|
||||
|
||||
Zuul Setup
|
||||
----------
|
||||
|
||||
At minimum you need to provide **zuul.conf** and **main.yaml** placed
|
||||
in **/etc/zuul/**. The following example uses the builtin gearman
|
||||
service in Zuul, and a connection to Gerrit.
|
||||
|
||||
**zuul.conf**::
|
||||
|
||||
[scheduler]
|
||||
tenant_config=/etc/zuul/main.yaml
|
||||
|
||||
[gearman_server]
|
||||
start=true
|
||||
|
||||
[gearman]
|
||||
server=127.0.0.1
|
||||
|
||||
[connection gerrit]
|
||||
driver=gerrit
|
||||
server=git.example.com
|
||||
port=29418
|
||||
baseurl=https://git.example.com/gerrit/
|
||||
user=zuul
|
||||
sshkey=/home/zuul/.ssh/id_rsa
|
||||
|
||||
See :ref:`components` and :ref:`connections` for more details.
|
||||
|
||||
The following tells Zuul to read its configuration from and operate on
|
||||
the *example-project* project:
|
||||
|
||||
**main.yaml**::
|
||||
|
||||
- tenant:
|
||||
name: example-tenant
|
||||
source:
|
||||
gerrit:
|
||||
untrusted-projects:
|
||||
- example-project
|
||||
|
||||
Starting Zuul
|
||||
-------------
|
||||
|
||||
You can run any zuul process with the **-d** option to make it not
|
||||
daemonize. It's a good idea at first to confirm there's no issues with
|
||||
your configuration.
|
||||
|
||||
To start, simply run::
|
||||
|
||||
zuul-scheduler
|
||||
|
||||
Once run you should have two zuul-scheduler processes (if using the
|
||||
built-in gearman server, or one process otherwise).
|
||||
|
||||
Before Zuul can run any jobs, it needs to load its configuration, most
|
||||
of which is in the git repositories that Zuul operates on. Start an
|
||||
executor to allow zuul to do that::
|
||||
|
||||
zuul-executor
|
||||
|
||||
Zuul should now be able to read its configuration from the configured
|
||||
repo and process any jobs defined therein.
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
You can use telnet to connect to gearman to check which Zuul
|
||||
components are online::
|
||||
|
||||
telnet <gearman_ip> 4730
|
||||
|
||||
Useful commands are **workers** and **status** which you can run by just
|
||||
typing those commands once connected to gearman.
|
||||
|
|
@ -0,0 +1,191 @@
|
|||
:title: Tenant Configuration
|
||||
|
||||
.. _tenant-config:
|
||||
|
||||
Tenant Configuration
|
||||
====================
|
||||
|
||||
After ``zuul.conf`` is configured, Zuul component servers will be able
|
||||
to start, but a tenant configuration is required in order for Zuul to
|
||||
perform any actions. The tenant configuration file specifies upon
|
||||
which projects Zuul should operate. These repositories are grouped
|
||||
into tenants. The configuration of each tenant is separate from the
|
||||
rest (no pipelines, jobs, etc are shared between them).
|
||||
|
||||
A project may appear in more than one tenant; this may be useful if
|
||||
you wish to use common job definitions across multiple tenants.
|
||||
|
||||
The tenant configuration file is specified by the
|
||||
:attr:`scheduler.tenant_config` setting in ``zuul.conf``. It is a
|
||||
YAML file which, like other Zuul configuration files, is a list of
|
||||
configuration objects, though only one type of object is supported:
|
||||
``tenant``.
|
||||
|
||||
Tenant
|
||||
------
|
||||
|
||||
A tenant is a collection of projects which share a Zuul
|
||||
configuration. An example tenant definition is:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- tenant:
|
||||
name: my-tenant
|
||||
max-nodes-per-job: 5
|
||||
exclude-unprotected-branches: false
|
||||
source:
|
||||
gerrit:
|
||||
config-projects:
|
||||
- common-config
|
||||
- shared-jobs:
|
||||
include: job
|
||||
untrusted-projects:
|
||||
- zuul-jobs:
|
||||
shadow: common-config
|
||||
- project1
|
||||
- project2:
|
||||
exclude-unprotected-branches: true
|
||||
|
||||
.. attr:: tenant
|
||||
|
||||
The following attributes are supported:
|
||||
|
||||
.. attr:: name
|
||||
:required:
|
||||
|
||||
The name of the tenant. This may appear in URLs, paths, and
|
||||
monitoring fields, and so should be restricted to URL friendly
|
||||
characters (ASCII letters, numbers, hyphen and underscore) and
|
||||
you should avoid changing it unless necessary.
|
||||
|
||||
.. attr:: source
|
||||
:required:
|
||||
|
||||
A dictionary of sources to consult for projects. A tenant may
|
||||
contain projects from multiple sources; each of those sources
|
||||
must be listed here, along with the projects it supports. The
|
||||
name of a :ref:`connection<connections>` is used as the
|
||||
dictionary key (e.g. ``gerrit`` in the example above), and the
|
||||
value is a further dictionary containing the keys below.
|
||||
|
||||
The next two attributes, **config-projects** and
|
||||
**untrusted-projects** provide the bulk of the information for
|
||||
tenant configuration. They list all of the projects upon which
|
||||
Zuul will act.
|
||||
|
||||
The order of the projects listed in a tenant is important. A job
|
||||
which is defined in one project may not be redefined in another
|
||||
project; therefore, once a job appears in one project, a project
|
||||
listed later will be unable to define a job with that name.
|
||||
Further, some aspects of project configuration (such as the merge
|
||||
mode) may only be set on the first appearance of a project
|
||||
definition.
|
||||
|
||||
Zuul loads the configuration from all **config-projects** in the
|
||||
order listed, followed by all **untrusted-projects** in order.
|
||||
|
||||
.. attr:: config-projects
|
||||
|
||||
A list of projects to be treated as :term:`config projects
|
||||
<config-project>` in this tenant. The jobs in a config project
|
||||
are trusted, which means they run with extra privileges, do not
|
||||
have their configuration dynamically loaded for proposed
|
||||
changes, and Zuul config files are only searched for in the
|
||||
``master`` branch.
|
||||
|
||||
The items in the list follow the same format described in
|
||||
**untrusted-projects**.
|
||||
|
||||
.. attr:: untrusted-projects
|
||||
|
||||
A list of projects to be treated as untrusted in this tenant.
|
||||
An :term:`untrusted-project` is the typical project operated on
|
||||
by Zuul. Their jobs run in a more restrictive environment, they
|
||||
may not define pipelines, their configuration dynamically
|
||||
changes in response to proposed changes, and Zuul will read
|
||||
configuration files in all of their branches.
|
||||
|
||||
.. attr:: <project>
|
||||
|
||||
The items in the list may either be simple string values of
|
||||
the project names, or a dictionary with the project name as
|
||||
key and the following values:
|
||||
|
||||
.. attr:: include
|
||||
|
||||
Normally Zuul will load all of the :ref:`configuration-items`
|
||||
appropriate for the type of project (config or untrusted)
|
||||
in question. However, if you only want to load some
|
||||
items, the **include** attribute can be used to specify
|
||||
that *only* the specified items should be loaded.
|
||||
Supplied as a string, or a list of strings.
|
||||
|
||||
The following **configuration items** are recognized:
|
||||
|
||||
* pipeline
|
||||
* job
|
||||
* semaphore
|
||||
* project
|
||||
* project-template
|
||||
* nodeset
|
||||
* secret
|
||||
|
||||
.. attr:: exclude
|
||||
|
||||
A list of **configuration items** that should not be loaded.
|
||||
|
||||
.. attr:: shadow
|
||||
|
||||
A list of projects which this project is permitted to
|
||||
shadow. Normally, only one project in Zuul may contain
|
||||
definitions for a given job. If a project earlier in the
|
||||
configuration defines a job which a later project
|
||||
redefines, the later definition is considered an error and
|
||||
is not permitted. The **shadow** attribute of a project
|
||||
indicates that job definitions in this project which
|
||||
conflict with the named projects should be ignored, and
|
||||
those in the named project should be used instead. The
|
||||
named projects must still appear earlier in the
|
||||
configuration. In the example above, if a job definition
|
||||
appears in both the ``common-config`` and ``zuul-jobs``
|
||||
projects, the definition in ``common-config`` will be
|
||||
used.
|
||||
|
||||
.. attr:: exclude-unprotected-branches
|
||||
|
||||
Define if unprotected github branches should be
|
||||
processed. Defaults to the tenant wide setting of
|
||||
exclude-unprotected-branches.
|
||||
|
||||
.. attr:: max-nodes-per-job
|
||||
:default: 5
|
||||
|
||||
The maximum number of nodes a job can request. A value of
|
||||
'-1' value removes the limit.
|
||||
|
||||
.. attr:: max-job-timeout
|
||||
:default: 10800
|
||||
|
||||
The maximum timeout for jobs. A value of '-1' value removes the limit.
|
||||
|
||||
.. attr:: exclude-unprotected-branches
|
||||
:default: false
|
||||
|
||||
When using a branch and pull model on a shared GitHub repository
|
||||
there are usually one or more protected branches which are gated
|
||||
and a dynamic number of personal/feature branches which are the
|
||||
source for the pull requests. These branches can potentially
|
||||
include broken Zuul config and therefore break the global tenant
|
||||
wide configuration. In order to deal with this Zuul's operations
|
||||
can be limited to the protected branches which are gated. This
|
||||
is a tenant wide setting and can be overridden per project.
|
||||
This currently only affects GitHub projects.
|
||||
|
||||
.. attr:: default-parent
|
||||
:default: base
|
||||
|
||||
If a job is defined without an explicit :attr:`job.parent`
|
||||
attribute, this job will be configured as the job's parent.
|
||||
This allows an administrator to configure a default base job to
|
||||
implement local policies such as node setup and artifact
|
||||
publishing.
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
:title: Zuul Client
|
||||
|
||||
Zuul Client
|
||||
===========
|
||||
|
||||
Zuul includes a simple command line client that may be used by
|
||||
administrators to affect Zuul's behavior while running. It must be
|
||||
run on a host that has access to the Gearman server (e.g., locally on
|
||||
the Zuul host).
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The client uses the same zuul.conf file as the server, and will look
|
||||
for it in the same locations if not specified on the command line.
|
||||
|
||||
Usage
|
||||
-----
|
||||
The general options that apply to all subcommands are:
|
||||
|
||||
.. program-output:: zuul --help
|
||||
|
||||
The following subcommands are supported:
|
||||
|
||||
Enqueue
|
||||
^^^^^^^
|
||||
.. program-output:: zuul enqueue --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul enqueue --trigger gerrit --pipeline check --project example_project --change 12345,1
|
||||
|
||||
Note that the format of change id is <number>,<patchset>.
|
||||
|
||||
Promote
|
||||
^^^^^^^
|
||||
.. program-output:: zuul promote --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul promote --pipeline check --changes 12345,1 13336,3
|
||||
|
||||
Note that the format of changes id is <number>,<patchset>.
|
||||
|
||||
Show
|
||||
^^^^
|
||||
.. program-output:: zuul show --help
|
||||
|
||||
Example::
|
||||
|
||||
zuul show running-jobs
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
:title: Zuul Cloner
|
||||
|
||||
Zuul Cloner
|
||||
===========
|
||||
|
||||
Zuul includes a simple command line client that may be used to clone
|
||||
repositories with Zuul references applied.
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Clone map
|
||||
'''''''''
|
||||
|
||||
By default, Zuul cloner will clone the project under ``basepath`` which
|
||||
would create sub directories whenever a project name contains slashes. Since
|
||||
you might want to finely tweak the final destination, a clone map lets you
|
||||
change the destination on a per project basis. The configuration is done using
|
||||
a YAML file passed with ``-m``.
|
||||
|
||||
With a project hierarchy such as::
|
||||
|
||||
project
|
||||
thirdparty/plugins/plugin1
|
||||
|
||||
You might want to get ``project`` straight in the base path, the clone map would be::
|
||||
|
||||
clonemap:
|
||||
- name: 'project'
|
||||
dest: '.'
|
||||
|
||||
Then to strip out ``thirdparty`` such that the plugins land under the
|
||||
``/plugins`` directory of the basepath, you can use regex and capturing
|
||||
groups::
|
||||
|
||||
clonemap:
|
||||
- name: 'project'
|
||||
dest: '.'
|
||||
- name: 'thirdparty/(plugins/.*)'
|
||||
dest: '\1'
|
||||
|
||||
The resulting workspace will contains::
|
||||
|
||||
project -> ./
|
||||
thirdparty/plugins/plugin1 -> ./plugins/plugin1
|
||||
|
||||
|
||||
Zuul parameters
|
||||
'''''''''''''''
|
||||
|
||||
The Zuul cloner reuses Zuul parameters such as ZUUL_BRANCH, ZUUL_REF or
|
||||
ZUUL_PROJECT. It will attempt to load them from the environment variables or
|
||||
you can pass them as parameters (in which case it will override the
|
||||
environment variable if it is set). The matching command line parameters use
|
||||
the ``zuul`` prefix hence ZUUL_REF can be passed to the cloner using
|
||||
``--zuul-ref``.
|
||||
|
||||
Usage
|
||||
-----
|
||||
The general options that apply are:
|
||||
|
||||
.. program-output:: zuul-cloner --help
|
||||
|
||||
|
||||
Ref lookup order
|
||||
''''''''''''''''
|
||||
|
||||
The Zuul cloner will attempt to lookup references in this order:
|
||||
|
||||
1) Zuul reference for the indicated branch
|
||||
2) Zuul reference for the master branch
|
||||
3) The tip of the indicated branch
|
||||
4) The tip of the master branch
|
||||
|
||||
The "indicated branch" is one of the following:
|
||||
|
||||
A) The project-specific override branch (from project_branches arg)
|
||||
B) The user specified branch (from the branch arg)
|
||||
C) ZUUL_BRANCH (from the zuul_branch arg)
|
||||
|
||||
Clone order
|
||||
-----------
|
||||
|
||||
When cloning repositories, the destination folder should not exist or
|
||||
``git clone`` will complain. This happens whenever cloning a sub project
|
||||
before its parent project. For example::
|
||||
|
||||
zuul-cloner project/plugins/plugin1 project
|
||||
|
||||
Will create the directory ``project`` when cloning the plugin. The
|
||||
cloner processes the clones in the order supplied, so you should swap the
|
||||
projects::
|
||||
|
||||
zuul-cloner project project/plugins/plugin1
|
||||
|
||||
Cached repositories
|
||||
-------------------
|
||||
|
||||
The ``--cache-dir`` option can be used to reduce network traffic by
|
||||
cloning from a local repository which may not be up to date.
|
||||
|
||||
If the ``--cache-dir`` option is supplied, zuul-cloner will start by
|
||||
cloning any projects it processes from those found in that directory.
|
||||
The URL of origin remote of the resulting clone will be reset to use
|
||||
the ``git_base_url`` and then the remote will be updated so that the
|
||||
repository has all the information in the upstream repository.
|
||||
|
||||
The default for ``--cache-dir`` is taken from the environment variable
|
||||
``ZUUL_CACHE_DIR``. A value provided explicitly on the command line
|
||||
overrides the environment variable setting.
|
||||
|
|
@ -25,10 +25,21 @@ sys.path.insert(0, os.path.abspath('../..'))
|
|||
|
||||
# Add any Sphinx extension module names here, as strings. They can be extensions
|
||||
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
|
||||
extensions = [ 'sphinxcontrib.blockdiag', 'sphinxcontrib.programoutput' ]
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
'sphinx_autodoc_typehints',
|
||||
'sphinx.ext.graphviz',
|
||||
'sphinxcontrib.blockdiag',
|
||||
'sphinxcontrib.programoutput',
|
||||
'zuul_sphinx',
|
||||
'zuul.sphinx.ansible',
|
||||
'zuul.sphinx.zuul',
|
||||
]
|
||||
#extensions = ['sphinx.ext.intersphinx']
|
||||
#intersphinx_mapping = {'python': ('http://docs.python.org/2.7', None)}
|
||||
|
||||
primary_domain = 'zuuldoc'
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
|
||||
|
|
@ -84,12 +95,14 @@ pygments_style = 'sphinx'
|
|||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
# a list of builtin themes.
|
||||
html_theme = 'default'
|
||||
#html_theme = 'alabaster'
|
||||
|
||||
# Theme options are theme-specific and customize the look and feel of a theme
|
||||
# further. For a list of options available for each theme, see the
|
||||
# documentation.
|
||||
#html_theme_options = {}
|
||||
html_theme_options = {
|
||||
'show_related': True
|
||||
}
|
||||
|
||||
# Add any paths that contain custom themes here, relative to this directory.
|
||||
#html_theme_path = []
|
||||
|
|
|
|||
|
|
@ -1,101 +0,0 @@
|
|||
:title: Connections
|
||||
|
||||
.. _connections:
|
||||
|
||||
Connections
|
||||
===========
|
||||
|
||||
zuul coordinates talking to multiple different systems via the concept
|
||||
of connections. A connection is listed in the :ref:`zuulconf` file and is
|
||||
then referred to from the :ref:`layoutyaml`. This makes it possible to
|
||||
receive events from gerrit via one connection and post results from another
|
||||
connection that may report back as a different user.
|
||||
|
||||
Gerrit
|
||||
------
|
||||
|
||||
Create a connection with gerrit.
|
||||
|
||||
**driver=gerrit**
|
||||
|
||||
**server**
|
||||
FQDN of Gerrit server.
|
||||
``server=review.example.com``
|
||||
|
||||
**port**
|
||||
Optional: Gerrit server port.
|
||||
``port=29418``
|
||||
|
||||
**baseurl**
|
||||
Optional: path to Gerrit web interface. Defaults to ``https://<value
|
||||
of server>/``. ``baseurl=https://review.example.com/review_site/``
|
||||
|
||||
**user**
|
||||
User name to use when logging into above server via ssh.
|
||||
``user=zuul``
|
||||
|
||||
**sshkey**
|
||||
Path to SSH key to use when logging into above server.
|
||||
``sshkey=/home/zuul/.ssh/id_rsa``
|
||||
|
||||
**keepalive**
|
||||
Optional: Keepalive timeout, 0 means no keepalive.
|
||||
``keepalive=60``
|
||||
|
||||
**strip_branch_ref**
|
||||
Optional: Earlier versions of Gerrit reported branch refs in the
|
||||
form "master" and later forms as "refs/heads/master". Set this to 1
|
||||
to enable compatibility with the earlier form by stripping the
|
||||
"refs/heads/" portion of the ref.
|
||||
``strip_branch_ref=1``
|
||||
|
||||
Gerrit Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Zuul will need access to a Gerrit user.
|
||||
|
||||
Create an SSH keypair for Zuul to use if there isn't one already, and
|
||||
create a Gerrit user with that key::
|
||||
|
||||
cat ~/id_rsa.pub | ssh -p29418 gerrit.example.com gerrit create-account --ssh-key - --full-name Jenkins jenkins
|
||||
|
||||
Give that user whatever permissions will be needed on the projects you
|
||||
want Zuul to gate. For instance, you may want to grant ``Verified
|
||||
+/-1`` and ``Submit`` to the user. Additional categories or values may
|
||||
be added to Gerrit. Zuul is very flexible and can take advantage of
|
||||
those.
|
||||
|
||||
SMTP
|
||||
----
|
||||
|
||||
**driver=smtp**
|
||||
|
||||
**server**
|
||||
SMTP server hostname or address to use.
|
||||
``server=localhost``
|
||||
|
||||
**port**
|
||||
Optional: SMTP server port.
|
||||
``port=25``
|
||||
|
||||
**default_from**
|
||||
Who the email should appear to be sent from when emailing the report.
|
||||
This can be overridden by individual pipelines.
|
||||
``default_from=zuul@example.com``
|
||||
|
||||
**default_to**
|
||||
Who the report should be emailed to by default.
|
||||
This can be overridden by individual pipelines.
|
||||
``default_to=you@example.com``
|
||||
|
||||
SQL
|
||||
----
|
||||
|
||||
Only one connection per a database is permitted.
|
||||
|
||||
**driver=sql**
|
||||
|
||||
**dburi**
|
||||
Database connection information in the form of a URI understood by
|
||||
sqlalchemy. eg http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#database-urls
|
||||
``dburi=mysql://user:pass@localhost/db``
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
Ansible Integration
|
||||
===================
|
||||
|
||||
Zuul contains Ansible modules and plugins to control the execution of Ansible
|
||||
Job content. These break down into two basic categories.
|
||||
|
||||
* Restricted Execution on Executors
|
||||
* Build Log Support
|
||||
|
||||
Restricted Execution
|
||||
--------------------
|
||||
|
||||
Zuul runs ``ansible-playbook`` on executors to run job content on nodes. While
|
||||
the intent is that content is run on the remote nodes, Ansible is a flexible
|
||||
system that allows delegating actions to ``localhost``, and also reading and
|
||||
writing files. These actions can be desirable and necessary for actions such
|
||||
as fetching log files or build artifacts, but could also be used as a vector
|
||||
to attack the executor.
|
||||
|
||||
For that reason Zuul implements a set of Ansible action plugins and lookup
|
||||
plugins that override and intercept task execution during untrusted playbook
|
||||
execution to ensure local actions are not executed or that for operations that
|
||||
are desirable to allow locally that they only interact with files in the zuul
|
||||
work directory.
|
||||
|
||||
.. autoclass:: zuul.ansible.action.normal.ActionModule
|
||||
:members:
|
||||
|
||||
Build Log Support
|
||||
-----------------
|
||||
|
||||
Zuul provides realtime build log streaming to end users so that users can
|
||||
watch long-running jobs in progress. As jobs may be written that execute a
|
||||
shell script that could run for a long time, additional effort is expended
|
||||
to stream stdout and stderr of shell tasks as they happen rather than waiting
|
||||
for the command to finish.
|
||||
|
||||
Zuul contains a modified version of the :ansible:module:`command`
|
||||
that starts a log streaming daemon on the build node.
|
||||
|
||||
.. automodule:: zuul.ansible.library.command
|
||||
|
||||
All jobs run with the :py:mod:`zuul.ansible.callback.zuul_stream` callback
|
||||
plugin enabled, which writes the build log to a file so that the
|
||||
:py:class:`zuul.lib.log_streamer.LogStreamer` can provide the data on demand
|
||||
over the finger protocol. Finally, :py:class:`zuul.web.LogStreamingHandler`
|
||||
exposes that log stream over a websocket connection as part of
|
||||
:py:class:`zuul.web.ZuulWeb`.
|
||||
|
||||
.. autoclass:: zuul.ansible.callback.zuul_stream.CallbackModule
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.lib.log_streamer.LogStreamer
|
||||
.. autoclass:: zuul.web.LogStreamingHandler
|
||||
.. autoclass:: zuul.web.ZuulWeb
|
||||
|
||||
In addition to real-time streaming, Zuul also installs another callback module,
|
||||
:py:mod:`zuul.ansible.callback.zuul_json.CallbackModule` that collects all
|
||||
of the information about a given run into a json file which is written to the
|
||||
work dir so that it can be published along with build logs. Since the streaming
|
||||
log is by necessity a single text stream, choices have to be made for
|
||||
readability about what data is shown and what is not shown. The json log file
|
||||
is intended to allow for a richer more interactive set of data to be displayed
|
||||
to the user.
|
||||
|
||||
.. autoclass:: zuul.ansible.callback.zuul_json.CallbackModule
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
Data Model
|
||||
==========
|
||||
|
||||
It all starts with the :py:class:`~zuul.model.Pipeline`. A Pipeline is the
|
||||
basic organizational structure that everything else hangs off.
|
||||
|
||||
.. autoclass:: zuul.model.Pipeline
|
||||
|
||||
Pipelines have a configured
|
||||
:py:class:`~zuul.manager.PipelineManager` which controlls how
|
||||
the :py:class:`Ref <zuul.model.Ref>` objects are enqueued and
|
||||
processed.
|
||||
|
||||
There are currently two,
|
||||
:py:class:`~zuul.manager.dependent.DependentPipelineManager` and
|
||||
:py:class:`~zuul.manager.independent.IndependentPipelineManager`
|
||||
|
||||
.. autoclass:: zuul.manager.PipelineManager
|
||||
.. autoclass:: zuul.manager.dependent.DependentPipelineManager
|
||||
.. autoclass:: zuul.manager.independent.IndependentPipelineManager
|
||||
|
||||
A :py:class:`~zuul.model.Pipeline` has one or more
|
||||
:py:class:`~zuul.model.ChangeQueue` objects.
|
||||
|
||||
.. autoclass:: zuul.model.ChangeQueue
|
||||
|
||||
A :py:class:`~zuul.model.Job` represents the definition of what to do. A
|
||||
:py:class:`~zuul.model.Build` represents a single run of a
|
||||
:py:class:`~zuul.model.Job`. A :py:class:`~zuul.model.JobGraph` is used to
|
||||
encapsulate the dependencies between one or more :py:class:`~zuul.model.Job`
|
||||
objects.
|
||||
|
||||
.. autoclass:: zuul.model.Job
|
||||
.. autoclass:: zuul.model.JobGraph
|
||||
.. autoclass:: zuul.model.Build
|
||||
|
||||
The :py:class:`~zuul.manager.base.PipelineManager` enqueues each
|
||||
:py:class:`Ref <zuul.model.Ref>` into the
|
||||
:py:class:`~zuul.model.ChangeQueue` in a :py:class:`~zuul.model.QueueItem`.
|
||||
|
||||
.. autoclass:: zuul.model.QueueItem
|
||||
|
||||
As the Changes are processed, each :py:class:`~zuul.model.Build` is put into
|
||||
a :py:class:`~zuul.model.BuildSet`
|
||||
|
||||
.. autoclass:: zuul.model.BuildSet
|
||||
|
||||
Changes
|
||||
~~~~~~~
|
||||
|
||||
.. autoclass:: zuul.model.Change
|
||||
.. autoclass:: zuul.model.Ref
|
||||
|
||||
Filters
|
||||
~~~~~~~
|
||||
|
||||
.. autoclass:: zuul.model.RefFilter
|
||||
.. autoclass:: zuul.model.EventFilter
|
||||
|
||||
|
||||
Tenants
|
||||
~~~~~~~
|
||||
|
||||
An abide is a collection of tenants.
|
||||
|
||||
.. autoclass:: zuul.model.Tenant
|
||||
.. autoclass:: zuul.model.UnparsedAbideConfig
|
||||
.. autoclass:: zuul.model.UnparsedTenantConfig
|
||||
|
||||
Other Global Objects
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: zuul.model.Project
|
||||
.. autoclass:: zuul.model.Layout
|
||||
.. autoclass:: zuul.model.RepoFiles
|
||||
.. autoclass:: zuul.model.Worker
|
||||
.. autoclass:: zuul.model.TriggerEvent
|
||||
|
|
@ -0,0 +1,225 @@
|
|||
Documentation
|
||||
=============
|
||||
|
||||
This is a brief style guide for Zuul documentation.
|
||||
|
||||
ReStructuredText Conventions
|
||||
----------------------------
|
||||
|
||||
Code Blocks
|
||||
~~~~~~~~~~~
|
||||
|
||||
When showing a YAML example, use the ``.. code-block:: yaml``
|
||||
directive so that the sample appears as a code block with the correct
|
||||
syntax highlighting.
|
||||
|
||||
Literal Values
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Filenames and literal values (such as when we instruct a user to type
|
||||
a specific string into a configuration file) should use the RST
|
||||
````literal```` syntax.
|
||||
|
||||
YAML supports boolean values expressed with or without an initial
|
||||
capital letter. In examples and documentation, use ``true`` and
|
||||
``false`` in lowercase type because the resulting YAML is easier for
|
||||
users to type and read.
|
||||
|
||||
Terminology
|
||||
~~~~~~~~~~~
|
||||
|
||||
Zuul employs some specialized terminology. To help users become
|
||||
acquainted with it, we employ a glossary. Observe the following:
|
||||
|
||||
* Specialized terms should have entries in the glossary.
|
||||
|
||||
* If the term is being defined in the text, don't link to the glossary
|
||||
(that would be redundant), but do emphasize it with ``*italics*``
|
||||
the first time it appears in that definition. Subsequent uses
|
||||
within the same subsection should be in regular type.
|
||||
|
||||
* If it's being used (but not defined) in the text, link the first
|
||||
usage within a subsection to the glossary using the ``:term:`` role,
|
||||
but subsequent uses should be in regular type.
|
||||
|
||||
* Be cognizant of how readers may jump to link targets within the
|
||||
text, so be liberal in considering that once you cross a link
|
||||
target, you may be in a new "subsection" for the above guideline.
|
||||
|
||||
|
||||
Zuul Sphinx Directives
|
||||
----------------------
|
||||
|
||||
The following extra Sphinx directives are available in the ``zuul``
|
||||
domain. The ``zuul`` domain is configured as the default domain, so the
|
||||
``zuul:`` prefix may be omitted.
|
||||
|
||||
zuul:attr::
|
||||
~~~~~~~~~~~
|
||||
|
||||
This should be used when documenting Zuul configuration attributes.
|
||||
Zuul configuration is heavily hierarchical, and this directive
|
||||
facilitates documenting these by emphasising the hierarchy as
|
||||
appropriate. It will annotate each configuration attribute with a
|
||||
nice header with its own unique hyperlink target. It displays the
|
||||
entire hierarchy of the attribute, but emphasises the last portion
|
||||
(i.e., the field being documented).
|
||||
|
||||
To use the hierarchical features, simply nest with indentation in the
|
||||
normal RST manner.
|
||||
|
||||
It supports the ``required`` and ``default`` options and will annotate
|
||||
the header appropriately. Example:
|
||||
|
||||
.. code-block:: rst
|
||||
|
||||
.. attr:: foo
|
||||
|
||||
Some text about ``foo``.
|
||||
|
||||
.. attr:: bar
|
||||
:required:
|
||||
:default: 42
|
||||
|
||||
Text about ``foo.bar``.
|
||||
|
||||
.. attr:: foo
|
||||
:noindex:
|
||||
|
||||
Some text about ``foo``.
|
||||
|
||||
.. attr:: bar
|
||||
:noindex:
|
||||
:required:
|
||||
:default: 42
|
||||
|
||||
Text about ``foo.bar``.
|
||||
|
||||
zuul:value::
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Similar to zuul:attr, but used when documenting a literal value of an
|
||||
attribute.
|
||||
|
||||
.. code-block:: rst
|
||||
|
||||
.. attr:: foo
|
||||
|
||||
Some text about foo. It supports the following values:
|
||||
|
||||
.. value:: bar
|
||||
|
||||
One of the supported values for ``foo`` is ``bar``.
|
||||
|
||||
.. value:: baz
|
||||
|
||||
Another supported values for ``foo`` is ``baz``.
|
||||
|
||||
.. attr:: foo
|
||||
:noindex:
|
||||
|
||||
Some text about foo. It supports the following values:
|
||||
|
||||
.. value:: bar
|
||||
:noindex:
|
||||
|
||||
One of the supported values for ``foo`` is ``bar``.
|
||||
|
||||
.. value:: baz
|
||||
:noindex:
|
||||
|
||||
Another supported values for ``foo`` is ``baz``.
|
||||
|
||||
zuul:var::
|
||||
~~~~~~~~~~
|
||||
|
||||
Also similar to zuul:attr, but used when documenting an Ansible
|
||||
variable which is available to a job's playbook. In these cases, it's
|
||||
often necessary to indicate the variable may be an element of a list
|
||||
or dictionary, so this directive supports a ``type`` option. It also
|
||||
supports the ``hidden`` option so that complex data structure
|
||||
definitions may continue across sections. To use this, set the hidden
|
||||
option on a ``zuul:var::`` directive with the root of the data
|
||||
structure as the name. Example:
|
||||
|
||||
.. code-block:: rst
|
||||
|
||||
.. var:: foo
|
||||
|
||||
Foo is a dictionary with the following keys:
|
||||
|
||||
.. var:: items
|
||||
:type: list
|
||||
|
||||
Items is a list of dictionaries with the following keys:
|
||||
|
||||
.. var:: bar
|
||||
|
||||
Text about bar
|
||||
|
||||
Section Boundary
|
||||
|
||||
.. var:: foo
|
||||
:hidden:
|
||||
|
||||
.. var:: baz
|
||||
|
||||
Text about baz
|
||||
|
||||
.. End of code block; start example
|
||||
|
||||
.. var:: foo
|
||||
:noindex:
|
||||
|
||||
Foo is a dictionary with the following keys:
|
||||
|
||||
.. var:: items
|
||||
:noindex:
|
||||
:type: list
|
||||
|
||||
Items is a list of dictionaries with the following keys:
|
||||
|
||||
.. var:: bar
|
||||
:noindex:
|
||||
|
||||
Text about bar
|
||||
|
||||
Section Boundary
|
||||
|
||||
.. var:: foo
|
||||
:noindex:
|
||||
:hidden:
|
||||
|
||||
.. var:: baz
|
||||
:noindex:
|
||||
|
||||
Text about baz
|
||||
|
||||
.. End of example
|
||||
|
||||
Zuul Sphinx Roles
|
||||
-----------------
|
||||
|
||||
The following extra Sphinx roles are available. Use these within the
|
||||
text when referring to attributes, values, and variables defined with
|
||||
the directives above. Use these roles for the first appearance of an
|
||||
object within a subsection, but use the ````literal```` role in
|
||||
subsequent uses.
|
||||
|
||||
:zuul:attr:
|
||||
~~~~~~~~~~~
|
||||
|
||||
This creates a reference to the named attribute. Provide the fully
|
||||
qualified name (e.g., ``:attr:`pipeline.manager```)
|
||||
|
||||
:zuul:value:
|
||||
~~~~~~~~~~~~
|
||||
|
||||
This creates a reference to the named value. Provide the fully
|
||||
qualified name (e.g., ``:attr:`pipeline.manager.dependent```)
|
||||
|
||||
:zuul:var:
|
||||
~~~~~~~~~~
|
||||
|
||||
This creates a reference to the named variable. Provide the fully
|
||||
qualified name (e.g., ``:var:`zuul.executor.name```)
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
Drivers
|
||||
=======
|
||||
|
||||
Zuul provides an API for extending its functionality to interact with
|
||||
other systems.
|
||||
|
||||
.. autoclass:: zuul.driver.Driver
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.ConnectionInterface
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.SourceInterface
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.TriggerInterface
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.ReporterInterface
|
||||
:members:
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
Developer's Guide
|
||||
=================
|
||||
|
||||
This section contains information for Developers who wish to work on
|
||||
Zuul itself. This information is not necessary for the operation of
|
||||
Zuul, though advanced users may find it interesting.
|
||||
|
||||
.. autoclass:: zuul.scheduler.Scheduler
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
datamodel
|
||||
drivers
|
||||
triggers
|
||||
testing
|
||||
docs
|
||||
ansible
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
Testing
|
||||
=======
|
||||
|
||||
Zuul provides an extensive framework for performing functional testing
|
||||
on the system from end-to-end with major external components replaced
|
||||
by fakes for ease of use and speed.
|
||||
|
||||
Test classes that subclass :py:class:`~tests.base.ZuulTestCase` have
|
||||
access to a number of attributes useful for manipulating or inspecting
|
||||
the environment being simulated in the test:
|
||||
|
||||
.. autofunction:: tests.base.simple_layout
|
||||
|
||||
.. autoclass:: tests.base.ZuulTestCase
|
||||
:members:
|
||||
|
||||
.. autoclass:: tests.base.FakeGerritConnection
|
||||
:members:
|
||||
:inherited-members:
|
||||
|
||||
.. autoclass:: tests.base.FakeGearmanServer
|
||||
:members:
|
||||
|
||||
.. autoclass:: tests.base.RecordingExecutorServer
|
||||
:members:
|
||||
|
||||
.. autoclass:: tests.base.FakeBuild
|
||||
:members:
|
||||
|
||||
.. autoclass:: tests.base.BuildHistory
|
||||
:members:
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
Triggers
|
||||
========
|
||||
|
||||
Triggers must inherit from :py:class:`~zuul.trigger.BaseTrigger` and, at a minimum,
|
||||
implement the :py:meth:`~zuul.trigger.BaseTrigger.getEventFilters` method.
|
||||
|
||||
.. autoclass:: zuul.trigger.BaseTrigger
|
||||
:members:
|
||||
|
||||
Current list of triggers are:
|
||||
|
||||
.. autoclass:: zuul.driver.gerrit.gerrittrigger.GerritTrigger
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.timer.timertrigger.TimerTrigger
|
||||
:members:
|
||||
|
||||
.. autoclass:: zuul.driver.zuul.zuultrigger.ZuulTrigger
|
||||
:members:
|
||||
|
|
@ -1,471 +0,0 @@
|
|||
:title: Project Gating
|
||||
|
||||
Project Gating
|
||||
==============
|
||||
|
||||
Traditionally, many software development projects merge changes from
|
||||
developers into the repository, and then identify regressions
|
||||
resulting from those changes (perhaps by running a test suite with a
|
||||
continuous integration system such as Jenkins), followed by more
|
||||
patches to fix those bugs. When the mainline of development is
|
||||
broken, it can be very frustrating for developers and can cause lost
|
||||
productivity, particularly so when the number of contributors or
|
||||
contributions is large.
|
||||
|
||||
The process of gating attempts to prevent changes that introduce
|
||||
regressions from being merged. This keeps the mainline of development
|
||||
open and working for all developers, and only when a change is
|
||||
confirmed to work without disruption is it merged.
|
||||
|
||||
Many projects practice an informal method of gating where developers
|
||||
with mainline commit access ensure that a test suite runs before
|
||||
merging a change. With more developers, more changes, and more
|
||||
comprehensive test suites, that process does not scale very well, and
|
||||
is not the best use of a developer's time. Zuul can help automate
|
||||
this process, with a particular emphasis on ensuring large numbers of
|
||||
changes are tested correctly.
|
||||
|
||||
Zuul was designed to handle the workflow of the OpenStack project, but
|
||||
can be used with any project.
|
||||
|
||||
Testing in parallel
|
||||
-------------------
|
||||
|
||||
A particular focus of Zuul is ensuring correctly ordered testing of
|
||||
changes in parallel. A gating system should always test each change
|
||||
applied to the tip of the branch exactly as it is going to be merged.
|
||||
A simple way to do that would be to test one change at a time, and
|
||||
merge it only if it passes tests. That works very well, but if
|
||||
changes take a long time to test, developers may have to wait a long
|
||||
time for their changes to make it into the repository. With some
|
||||
projects, it may take hours to test changes, and it is easy for
|
||||
developers to create changes at a rate faster than they can be tested
|
||||
and merged.
|
||||
|
||||
Zuul's DependentPipelineManager allows for parallel execution of test
|
||||
jobs for gating while ensuring changes are tested correctly, exactly
|
||||
as if they had been tested one at a time. It does this by performing
|
||||
speculative execution of test jobs; it assumes that all jobs will
|
||||
succeed and tests them in parallel accordingly. If they do succeed,
|
||||
they can all be merged. However, if one fails, then changes that were
|
||||
expecting it to succeed are re-tested without the failed change. In
|
||||
the best case, as many changes as execution contexts are available may
|
||||
be tested in parallel and merged at once. In the worst case, changes
|
||||
are tested one at a time (as each subsequent change fails, changes
|
||||
behind it start again). In practice, the OpenStack project observes
|
||||
something closer to the best case.
|
||||
|
||||
For example, if a core developer approves five changes in rapid
|
||||
succession::
|
||||
|
||||
A, B, C, D, E
|
||||
|
||||
Zuul queues those changes in the order they were approved, and notes
|
||||
that each subsequent change depends on the one ahead of it merging:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
A <- B <- C <- D <- E;
|
||||
}
|
||||
|
||||
Zuul then starts immediately testing all of the changes in parallel.
|
||||
But in the case of changes that depend on others, it instructs the
|
||||
test system to include the changes ahead of it, with the assumption
|
||||
they pass. That means jobs testing change *B* include change *A* as
|
||||
well::
|
||||
|
||||
Jobs for A: merge change A, then test
|
||||
Jobs for B: merge changes A and B, then test
|
||||
Jobs for C: merge changes A, B and C, then test
|
||||
Jobs for D: merge changes A, B, C and D, then test
|
||||
Jobs for E: merge changes A, B, C, D and E, then test
|
||||
|
||||
Hence jobs triggered to tests A will only test A and ignore B, C, D:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
master -> A -> B -> C -> D -> E;
|
||||
group jobs_for_A {
|
||||
label = "Merged changes for A";
|
||||
master -> A;
|
||||
}
|
||||
group ignored_to_test_A {
|
||||
label = "Ignored changes";
|
||||
color = "lightgray";
|
||||
B -> C -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
The jobs for E would include the whole dependency chain: A, B, C, D, and E.
|
||||
E will be tested assuming A, B, C, and D passed:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
group jobs_for_E {
|
||||
label = "Merged changes for E";
|
||||
master -> A -> B -> C -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
If changes *A* and *B* pass tests (green), and *C*, *D*, and *E* fail (red):
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
A [color = lightgreen];
|
||||
B [color = lightgreen];
|
||||
C [color = pink];
|
||||
D [color = pink];
|
||||
E [color = pink];
|
||||
|
||||
master <- A <- B <- C <- D <- E;
|
||||
}
|
||||
|
||||
Zuul will merge change *A* followed by change *B*, leaving this queue:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
C [color = pink];
|
||||
D [color = pink];
|
||||
E [color = pink];
|
||||
|
||||
C <- D <- E;
|
||||
}
|
||||
|
||||
Since *D* was dependent on *C*, it is not clear whether *D*'s failure is the
|
||||
result of a defect in *D* or *C*:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
C [color = pink];
|
||||
D [label = "D\n?"];
|
||||
E [label = "E\n?"];
|
||||
|
||||
C <- D <- E;
|
||||
}
|
||||
|
||||
Since *C* failed, Zuul will report its failure and drop *C* from the queue,
|
||||
keeping D and E:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
D [label = "D\n?"];
|
||||
E [label = "E\n?"];
|
||||
|
||||
D <- E;
|
||||
}
|
||||
|
||||
This queue is the same as if two new changes had just arrived, so Zuul
|
||||
starts the process again testing *D* against the tip of the branch, and
|
||||
*E* against *D*:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
master -> D -> E;
|
||||
group jobs_for_D {
|
||||
label = "Merged changes for D";
|
||||
master -> D;
|
||||
}
|
||||
group ignored_to_test_D {
|
||||
label = "Skip";
|
||||
color = "lightgray";
|
||||
E;
|
||||
}
|
||||
}
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
group jobs_for_E {
|
||||
label = "Merged changes for E";
|
||||
master -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Cross Project Testing
|
||||
---------------------
|
||||
|
||||
When your projects are closely coupled together, you want to make sure
|
||||
changes entering the gate are going to be tested with the version of
|
||||
other projects currently enqueued in the gate (since they will
|
||||
eventually be merged and might introduce breaking features).
|
||||
|
||||
Such relationships can be defined in Zuul configuration by registering
|
||||
a job in a DependentPipeline of several projects. Whenever a change
|
||||
enters such a pipeline, it will create references for the other
|
||||
projects as well. As an example, given a main project ``acme`` and a
|
||||
plugin ``plugin`` you can define a job ``acme-tests`` which should be
|
||||
run for both projects:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
pipelines:
|
||||
- name: gate
|
||||
manager: DependentPipelineManager
|
||||
|
||||
projects::
|
||||
- name: acme
|
||||
gate:
|
||||
- acme-tests
|
||||
- name: plugin
|
||||
gate:
|
||||
- acme-tests # Register job again
|
||||
|
||||
Whenever a change enters the ``gate`` pipeline queue, Zuul creates a reference
|
||||
for it. For each subsequent change, an additional reference is created for the
|
||||
changes ahead in the queue. As a result, you will always be able to fetch the
|
||||
future state of your project dependencies for each change in the queue.
|
||||
|
||||
Based on the pipeline and project definitions above, three changes are
|
||||
inserted in the ``gate`` pipeline with the associated references:
|
||||
|
||||
======== ======= ====== =========
|
||||
Change Project Branch Zuul Ref.
|
||||
======== ======= ====== =========
|
||||
Change 1 acme master master/Z1
|
||||
Change 2 plugin stable stable/Z2
|
||||
Change 3 plugin master master/Z3
|
||||
======== ======= ====== =========
|
||||
|
||||
Since the changes enter a DependentPipelineManager pipeline, Zuul creates
|
||||
additional references:
|
||||
|
||||
====== ======= ========= =============================
|
||||
Change Project Zuul Ref. Description
|
||||
====== ======= ========= =============================
|
||||
1 acme master/Z1 acme master + change 1
|
||||
------ ------- --------- -----------------------------
|
||||
2 acme master/Z2 acme master + change 1
|
||||
2 plugin stable/Z2 plugin stable + change 2
|
||||
------ ------- --------- -----------------------------
|
||||
3 acme master/Z3 acme master + change 1
|
||||
3 plugin stable/Z3 plugin stable + change 2
|
||||
3 plugin master/Z3 plugin master + change 3
|
||||
====== ======= ========= =============================
|
||||
|
||||
In order to test change 3, you would clone both repositories and simply
|
||||
fetch the Z3 reference for each combination of project/branch you are
|
||||
interested in testing. For example, you could fetch ``acme`` with
|
||||
master/Z3 and ``plugin`` with master/Z3 and thus have ``acme`` with
|
||||
change 1 applied as the expected state for when Change 3 would merge.
|
||||
When your job fetches several repositories without changes ahead in the
|
||||
queue, they may not have a Z reference in which case you can just check
|
||||
out the branch.
|
||||
|
||||
|
||||
Cross Repository Dependencies
|
||||
-----------------------------
|
||||
|
||||
Zuul permits users to specify dependencies across repositories. Using
|
||||
a special header in Git commit messages, Users may specify that a
|
||||
change depends on another change in any repository known to Zuul.
|
||||
|
||||
Zuul's cross-repository dependencies (CRD) behave like a directed
|
||||
acyclic graph (DAG), like git itself, to indicate a one-way dependency
|
||||
relationship between changes in different git repositories. Change A
|
||||
may depend on B, but B may not depend on A.
|
||||
|
||||
To use them, include ``Depends-On: <gerrit-change-id>`` in the footer of
|
||||
a commit message. Use the full Change-ID ('I' + 40 characters).
|
||||
|
||||
|
||||
Dependent Pipeline
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When Zuul sees CRD changes, it serializes them in the usual manner when
|
||||
enqueuing them into a pipeline. This means that if change A depends on
|
||||
B, then when they are added to a dependent pipeline, B will appear first
|
||||
and A will follow:
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crd {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class greendot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
color = green,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
A_status [ class = greendot ]
|
||||
B_status [ class = greendot ]
|
||||
B_status -- A_status
|
||||
|
||||
'Change B\nChange-Id: Iabc' <- 'Change A\nDepends-On: Iabc'
|
||||
}
|
||||
|
||||
If tests for B fail, both B and A will be removed from the pipeline, and
|
||||
it will not be possible for A to merge until B does.
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
If changes with CRD do not share a change queue then Zuul is unable
|
||||
to enqueue them together, and the first will be required to merge
|
||||
before the second is enqueued.
|
||||
|
||||
Independent Pipeline
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When changes are enqueued into an independent pipeline, all of the
|
||||
related dependencies (both normal git-dependencies that come from parent
|
||||
commits as well as CRD changes) appear in a dependency graph, as in a
|
||||
dependent pipeline. This means that even in an independent pipeline,
|
||||
your change will be tested with its dependencies. So changes that were
|
||||
previously unable to be fully tested until a related change landed in a
|
||||
different repository may now be tested together from the start.
|
||||
|
||||
All of the changes are still independent (so you will note that the
|
||||
whole pipeline does not share a graph as in a dependent pipeline), but
|
||||
for each change tested, all of its dependencies are visually connected
|
||||
to it, and they are used to construct the git references that Zuul uses
|
||||
when testing.
|
||||
|
||||
When looking at this graph on the status page, you will note that the
|
||||
dependencies show up as grey dots, while the actual change tested shows
|
||||
up as red or green (depending on the jobs results):
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crdgrey {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class dot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
A_status [class = "dot", color = green]
|
||||
B_status [class = "dot", color = grey]
|
||||
B_status -- A_status
|
||||
|
||||
"Change B" <- "Change A\nDepends-On: B"
|
||||
}
|
||||
|
||||
This is to indicate that the grey changes are only there to establish
|
||||
dependencies. Even if one of the dependencies is also being tested, it
|
||||
will show up as a grey dot when used as a dependency, but separately and
|
||||
additionally will appear as its own red or green dot for its test.
|
||||
|
||||
|
||||
Multiple Changes
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
A Gerrit change ID may refer to multiple changes (on multiple branches
|
||||
of the same project, or even multiple projects). In these cases, Zuul
|
||||
will treat all of the changes with that change ID as dependencies. So
|
||||
if you say that change in project A Depends-On a change ID that has
|
||||
changes in two branches of project B, then when testing the change to
|
||||
project A, both project B changes will be applied, and when deciding
|
||||
whether the project A change can merge, both changes must merge ahead
|
||||
of it.
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crdmultirepos {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class greendot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
color = green,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
B_stable_status [ class = "greendot" ]
|
||||
B_master_status [ class = "greendot" ]
|
||||
A_status [ class = "greendot" ]
|
||||
B_stable_status -- B_master_status -- A_status
|
||||
|
||||
A [ label = "Repo A\nDepends-On: I123" ]
|
||||
group {
|
||||
orientation = portrait
|
||||
label = "Dependencies"
|
||||
color = "lightgray"
|
||||
|
||||
B_stable [ label = "Repo B\nChange-Id: I123\nBranch: stable" ]
|
||||
B_master [ label = "Repo B\nChange-Id: I123\nBranch: master" ]
|
||||
}
|
||||
B_master <- A
|
||||
B_stable <- A
|
||||
|
||||
}
|
||||
|
||||
A change may depend on more than one Gerrit change ID as well. So it
|
||||
is possible for a change in project A to depend on a change in project
|
||||
B and a change in project C. Simply add more ``Depends-On:`` lines to
|
||||
the commit message footer.
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crdmultichanges {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class greendot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
color = green,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
C_status [ class = "greendot" ]
|
||||
B_status [ class = "greendot" ]
|
||||
A_status [ class = "greendot" ]
|
||||
C_status -- B_status -- A_status
|
||||
|
||||
A [ label = "Repo A\nDepends-On: I123\nDepends-On: Iabc" ]
|
||||
group {
|
||||
orientation = portrait
|
||||
label = "Dependencies"
|
||||
color = "lightgray"
|
||||
|
||||
B [ label = "Repo B\nChange-Id: I123" ]
|
||||
C [ label = "Repo C\nChange-Id: Iabc" ]
|
||||
}
|
||||
B, C <- A
|
||||
}
|
||||
|
||||
Cycles
|
||||
~~~~~~
|
||||
|
||||
If a cycle is created by use of CRD, Zuul will abort its work very
|
||||
early. There will be no message in Gerrit and no changes that are part
|
||||
of the cycle will be enqueued into any pipeline. This is to protect
|
||||
Zuul from infinite loops.
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
.. _glossary:
|
||||
|
||||
Glossary
|
||||
========
|
||||
|
||||
.. glossary::
|
||||
:sorted:
|
||||
|
||||
base job
|
||||
|
||||
A job with no parent. A base job may only be defined in a
|
||||
:term:`config-project`. Multiple base jobs may be defined, but
|
||||
each tenant has a single default job which will be used as the
|
||||
parent of any job which does not specify one explicitly.
|
||||
|
||||
check
|
||||
|
||||
By convention, the name of a pipeline which performs pre-merge
|
||||
tests. Such a pipeline might be triggered by creating a new
|
||||
change or pull request. It may run with changes which have not
|
||||
yet seen any human review, so care must be taken in selecting
|
||||
the kinds of jobs to run, and what resources will be available
|
||||
to them in order to avoid misuse of the system or credential
|
||||
compromise.
|
||||
|
||||
config-project
|
||||
|
||||
One of two types of projects which may be specified by the
|
||||
administrator in the tenant config file. A config-project is
|
||||
primarily tasked with holding configuration information and job
|
||||
content for Zuul. Jobs which are defined in a config-project
|
||||
are run with elevated privileges, and all Zuul configuration
|
||||
items are available for use. It is expected that changes to
|
||||
config-projects will undergo careful scrutiny before being
|
||||
merged.
|
||||
|
||||
gate
|
||||
|
||||
By convention, the name of a pipeline which performs project
|
||||
gating. Such a pipeline might be triggered by a core team
|
||||
member approving a change or pull request. It should have a
|
||||
:value:`dependent <pipeline.manager.dependent>` pipeline manager
|
||||
so that it can combine and sequence changes as they are
|
||||
approved.
|
||||
|
||||
reporter
|
||||
|
||||
A reporter is a :ref:`pipeline attribute <reporters>` which
|
||||
describes the action performed when an item is dequeued after
|
||||
its jobs complete. Reporters are implemented by :ref:`drivers`
|
||||
so their actions may be quite varied. For example, a reporter
|
||||
might leave feedback in a remote system on a proposed change,
|
||||
send email, or store information in a database.
|
||||
|
||||
trusted execution context
|
||||
|
||||
Playbooks defined in a :term:`config-project` run in the
|
||||
*trusted* execution context. The trusted execution context has
|
||||
access to all Ansible features, including the ability to load
|
||||
custom Ansible modules.
|
||||
|
||||
untrusted execution context
|
||||
|
||||
Playbooks defined in an :term:`untrusted-project` run in the
|
||||
*untrusted* execution context. Playbooks run in the untrusted
|
||||
execution context are not permitted to load additional Ansible
|
||||
modules or access files outside of the restricted environment
|
||||
prepared for them by the executor. In addition to the
|
||||
bubblewrap environment applied to both execution contexts, in
|
||||
the untrusted context some standard Ansible modules are replaced
|
||||
with versions which prohibit some actions, including attempts to
|
||||
access files outside of the restricted execution context. These
|
||||
redundant protections are made as part of a defense-in-depth
|
||||
strategy.
|
||||
|
||||
untrusted-project
|
||||
|
||||
One of two types of projects which may be specified by the
|
||||
administrator in the tenant config file. An untrusted-project
|
||||
is one whose primary focus is not to operate Zuul, but rather it
|
||||
is one of the projects being tested or deployed. The Zuul
|
||||
configuration language available to these projects is somewhat
|
||||
restricted, and jobs defined in these projects run in a
|
||||
restricted execution environment since they may be operating on
|
||||
changes which have not yet undergone review.
|
||||
|
|
@ -1,33 +1,43 @@
|
|||
Zuul - A Project Gating System
|
||||
==============================
|
||||
|
||||
Zuul is a program that is used to gate the source code repository of a
|
||||
project so that changes are only merged if they pass tests.
|
||||
Zuul is a program that drives continuous integration, delivery, and
|
||||
deployment systems with a focus on project gating and interrelated
|
||||
projects.
|
||||
|
||||
The main component of Zuul is the scheduler. It receives events
|
||||
related to proposed changes, triggers tests based on those events, and
|
||||
reports back.
|
||||
Zuul's documentation is organized in three guides based on audience.
|
||||
If Zuul is being used to gate or drive automation around your project,
|
||||
read the :doc:`user/index` to find out how to configure Zuul. If you
|
||||
are installing or operating a Zuul system, you will also find the
|
||||
:doc:`admin/index` useful. If you want help make Zuul itself better,
|
||||
take a look at the :doc:`developer/index`.
|
||||
|
||||
If you are looking for the Edge routing service named Zuul that is
|
||||
related to Netflix, it can be found here:
|
||||
https://github.com/Netflix/zuul
|
||||
|
||||
If you are looking for the Javascript testing tool named Zuul, it
|
||||
can be found here:
|
||||
https://github.com/defunctzombie/zuul
|
||||
|
||||
Contents:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
quick-start
|
||||
gating
|
||||
connections
|
||||
triggers
|
||||
reporters
|
||||
zuul
|
||||
merger
|
||||
cloner
|
||||
launchers
|
||||
statsd
|
||||
client
|
||||
user/index
|
||||
admin/index
|
||||
developer/index
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
glossary
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`search`
|
||||
* :ref:`glossary`
|
||||
|
||||
|
|
|
|||
|
|
@ -1,385 +0,0 @@
|
|||
:title: Launchers
|
||||
|
||||
.. _Gearman: http://gearman.org/
|
||||
|
||||
.. _`Gearman Plugin`:
|
||||
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
|
||||
|
||||
.. _`Turbo-Hipster`:
|
||||
https://git.openstack.org/cgit/openstack/turbo-hipster/
|
||||
|
||||
.. _`Turbo-Hipster Documentation`:
|
||||
http://turbo-hipster.rtfd.org/
|
||||
|
||||
.. _FormPost: http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
|
||||
|
||||
.. _launchers:
|
||||
|
||||
Launchers
|
||||
=========
|
||||
|
||||
Zuul has a modular architecture for launching jobs. Currently, the
|
||||
only supported module interfaces with Gearman_. This design allows
|
||||
any system to run jobs for Zuul simply by interfacing with a Gearman
|
||||
server. The recommended way of integrating a new job-runner with Zuul
|
||||
is via this method.
|
||||
|
||||
If Gearman is unsuitable, Zuul may be extended with a new launcher
|
||||
module. Zuul makes very few assumptions about the interface to a
|
||||
launcher -- if it can trigger jobs, cancel them, and receive success
|
||||
or failure reports, it should be able to be used with Zuul. Patches
|
||||
to this effect are welcome.
|
||||
|
||||
Zuul Parameters
|
||||
---------------
|
||||
|
||||
Zuul will pass some parameters with every job it launches. These are
|
||||
for workers to be able to get the repositories into the state they are
|
||||
intended to be tested in. Builds can be triggered either by an action
|
||||
on a change or by a reference update. Both events share a common set
|
||||
of parameters and more specific parameters as follows:
|
||||
|
||||
Common parameters
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
**ZUUL_UUID**
|
||||
Zuul provided key to link builds with Gerrit events.
|
||||
**ZUUL_REF**
|
||||
Zuul provided ref that includes commit(s) to build.
|
||||
**ZUUL_COMMIT**
|
||||
The commit SHA1 at the head of ZUUL_REF.
|
||||
**ZUUL_PROJECT**
|
||||
The project that triggered this build.
|
||||
**ZUUL_PIPELINE**
|
||||
The Zuul pipeline that is building this job.
|
||||
**ZUUL_URL**
|
||||
The URL for the zuul server as configured in zuul.conf.
|
||||
A test runner may use this URL as the basis for fetching
|
||||
git commits.
|
||||
**BASE_LOG_PATH**
|
||||
zuul suggests a path to store and address logs. This is deterministic
|
||||
and hence useful for where you want workers to upload to a specific
|
||||
destination or need them to have a specific final URL you can link to
|
||||
in advanced. For changes it is:
|
||||
"last-two-digits-of-change/change-number/patchset-number".
|
||||
For reference updates it is: "first-two-digits-of-newrev/newrev"
|
||||
**LOG_PATH**
|
||||
zuul also suggests a unique path for logs to the worker. This is
|
||||
"BASE_LOG_PATH/pipeline-name/job-name/uuid"
|
||||
**ZUUL_VOTING**
|
||||
Whether Zuul considers this job voting or not. Note that if Zuul is
|
||||
reconfigured during the run, the voting status of a job may change
|
||||
and this value will be out of date. Values are '1' if voting, '0'
|
||||
otherwise.
|
||||
|
||||
Change related parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following additional parameters will only be provided for builds
|
||||
associated with changes (i.e., in response to patchset-created or
|
||||
comment-added events):
|
||||
|
||||
**ZUUL_BRANCH**
|
||||
The target branch for the change that triggered this build.
|
||||
**ZUUL_CHANGE**
|
||||
The Gerrit change ID for the change that triggered this build.
|
||||
**ZUUL_CHANGES**
|
||||
A caret character separated list of the changes upon which this build
|
||||
is dependent upon in the form of a colon character separated list
|
||||
consisting of project name, target branch, and revision ref.
|
||||
**ZUUL_CHANGE_IDS**
|
||||
All of the Gerrit change IDs that are included in this build (useful
|
||||
when the DependentPipelineManager combines changes for testing).
|
||||
**ZUUL_PATCHSET**
|
||||
The Gerrit patchset number for the change that triggered this build.
|
||||
|
||||
Reference updated parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following additional parameters will only be provided for
|
||||
post-merge (ref-updated) builds:
|
||||
|
||||
**ZUUL_OLDREV**
|
||||
The SHA1 of the old revision at this ref (recall the ref name is
|
||||
in ZUUL_REF).
|
||||
**ZUUL_NEWREV**
|
||||
The SHA1 of the new revision at this ref (recall the ref name is
|
||||
in ZUUL_REF).
|
||||
**ZUUL_SHORT_OLDREV**
|
||||
The shortened (7 character) SHA1 of the old revision.
|
||||
**ZUUL_SHORT_NEWREV**
|
||||
The shortened (7 character) SHA1 of the new revision.
|
||||
|
||||
Unset revisions default to 00000000000000000000000000000000.
|
||||
|
||||
Examples:
|
||||
|
||||
When a reference is created::
|
||||
|
||||
ZUUL_OLDREV=00000000000000000000000000000000
|
||||
ZUUL_NEWREV=123456789abcdef123456789abcdef12
|
||||
ZUUL_SHORT_OLDREV=0000000
|
||||
ZUUL_SHORT_NEWREV=1234567
|
||||
|
||||
When a reference is deleted::
|
||||
|
||||
ZUUL_OLDREV=123456789abcdef123456789abcdef12
|
||||
ZUUL_NEWREV=00000000000000000000000000000000
|
||||
ZUUL_SHORT_OLDREV=1234567
|
||||
ZUUL_SHORT_NEWREV=0000000
|
||||
|
||||
And finally a reference being altered::
|
||||
|
||||
ZUUL_OLDREV=123456789abcdef123456789abcdef12
|
||||
ZUUL_NEWREV=abcdef123456789abcdef123456789ab
|
||||
ZUUL_SHORT_OLDREV=1234567
|
||||
ZUUL_SHORT_NEWREV=abcdef1
|
||||
|
||||
Your jobs can check whether the parameters are ``000000`` to act
|
||||
differently on each kind of event.
|
||||
|
||||
Swift parameters
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
If swift information has been configured for the job zuul will also
|
||||
provide signed credentials for the builder to upload results and
|
||||
assets into containers using the `FormPost`_ middleware.
|
||||
|
||||
Each zuul container/instruction set will contain each of the following
|
||||
parameters where $NAME is the ``name`` defined in the layout.
|
||||
|
||||
*SWIFT_$NAME_URL*
|
||||
The swift destination URL. This will be the entire URL including
|
||||
the AUTH, container and path prefix (folder).
|
||||
*SWIFT_$NAME_HMAC_BODY*
|
||||
The information signed in the HMAC body. The body is as follows::
|
||||
|
||||
PATH TO OBJECT PREFIX (excluding domain)
|
||||
BLANK LINE (zuul implements no form redirect)
|
||||
MAX FILE SIZE
|
||||
MAX FILE COUNT
|
||||
SIGNATURE EXPIRY
|
||||
|
||||
*SWIFT_$NAME_SIGNATURE*
|
||||
The HMAC body signed with the configured key.
|
||||
*SWIFT_$NAME_LOGSERVER_PREFIX*
|
||||
The URL to prepend to the object path when returning the results
|
||||
from a build.
|
||||
|
||||
Gearman
|
||||
-------
|
||||
|
||||
Gearman_ is a general-purpose protocol for distributing jobs to any
|
||||
number of workers. Zuul works with Gearman by sending specific
|
||||
information with job requests to Gearman, and expects certain
|
||||
information to be returned on completion. This protocol is described
|
||||
in `Zuul-Gearman Protocol`_.
|
||||
|
||||
In order for Zuul to run any jobs, you will need a running Gearman
|
||||
server. Zuul includes a Gearman server, and it is recommended that it
|
||||
be used as it supports the following features needed by Zuul:
|
||||
|
||||
* Canceling jobs in the queue (admin protocol command "cancel job").
|
||||
* Strict FIFO queue operation (gearmand's round-robin mode may be
|
||||
sufficient, but is untested).
|
||||
|
||||
To enable the built-in server, see the ``gearman_server`` section of
|
||||
``zuul.conf``. Be sure that the host allows connections from Zuul and
|
||||
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
|
||||
(as the Gearman protocol does not include any provision for
|
||||
authentication).
|
||||
|
||||
Gearman Jenkins Plugin
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
||||
by providing an interface between Jenkins and Gearman. In this
|
||||
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
||||
distribute those jobs to any number of Jenkins systems (including
|
||||
multiple Jenkins masters).
|
||||
|
||||
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
||||
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
||||
configure it with the hostname or IP address of your Gearman server
|
||||
and the port on which it is listening (4730 by default). It will
|
||||
automatically register all known Jenkins jobs as functions that Zuul
|
||||
can invoke via Gearman.
|
||||
|
||||
Any number of masters can be configured in this way, and Gearman will
|
||||
distribute jobs to all of them as appropriate.
|
||||
|
||||
No special Jenkins job configuration is needed to support triggering
|
||||
by Zuul.
|
||||
|
||||
The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
|
||||
Jenkins build parameters, so they will be available for use in the job
|
||||
configuration as well as to the running job as environment variables.
|
||||
|
||||
Jenkins git plugin configuration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to test the correct build, configure the Jenkins Git SCM
|
||||
plugin as follows::
|
||||
|
||||
Source Code Management:
|
||||
Git
|
||||
Repositories:
|
||||
Repository URL: <your Gerrit or Zuul repository URL>
|
||||
Advanced:
|
||||
Refspec: ${ZUUL_REF}
|
||||
Branches to build:
|
||||
Branch Specifier: ${ZUUL_COMMIT}
|
||||
Advanced:
|
||||
Clean after checkout: True
|
||||
|
||||
That should be sufficient for a job that only builds a single project.
|
||||
If you have multiple interrelated projects (i.e., they share a Zuul
|
||||
Change Queue) that are built together, you may be able to configure
|
||||
the Git plugin to prepare them, or you may chose to use a shell script
|
||||
instead. As an example, the OpenStack project uses the following
|
||||
script to prepare the workspace for its integration testing:
|
||||
|
||||
https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh
|
||||
|
||||
Turbo Hipster Worker
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
As an alternative to Jenkins, `Turbo-Hipster`_ is a small python
|
||||
project designed specifically as a zuul job worker which can be
|
||||
registered with gearman as a job runner. Please see the
|
||||
`Turbo-Hipster Documentation`_ for details on how to set it up.
|
||||
|
||||
Zuul-Gearman Protocol
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This section is only relevant if you intend to implement a new kind of
|
||||
worker that runs jobs for Zuul via Gearman. If you just want to use
|
||||
Jenkins, see `Gearman Jenkins Plugin`_ instead.
|
||||
|
||||
The Zuul protocol as used with Gearman is as follows:
|
||||
|
||||
Starting Builds
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
To start a build, Zuul invokes a Gearman function with the following
|
||||
format:
|
||||
|
||||
build:FUNCTION_NAME
|
||||
|
||||
where **FUNCTION_NAME** is the name of the job that should be run. If
|
||||
the job should run on a specific node (or class of node), Zuul will
|
||||
instead invoke:
|
||||
|
||||
build:FUNCTION_NAME:NODE_NAME
|
||||
|
||||
where **NODE_NAME** is the name or class of node on which the job
|
||||
should be run. This can be specified by setting the ZUUL_NODE
|
||||
parameter in a parameter-function (see :ref:`includes` section in
|
||||
:ref:`zuulconf`).
|
||||
|
||||
Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
|
||||
encoded in JSON format as the argument included with the
|
||||
SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
|
||||
ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
|
||||
an added Gearman parameter with GRAB_JOB_UNIQ.
|
||||
|
||||
When a Gearman worker starts running a job for Zuul, it should
|
||||
immediately send a WORK_DATA packet with the following information
|
||||
encoded in JSON format:
|
||||
|
||||
**name**
|
||||
The name of the job.
|
||||
|
||||
**number**
|
||||
The build number (unique to this job).
|
||||
|
||||
**manager**
|
||||
A unique identifier associated with the Gearman worker that can
|
||||
abort this build. See `Stopping Builds`_ for more information.
|
||||
|
||||
**url** (optional)
|
||||
The URL with the status or results of the build. Will be used in
|
||||
the status page and the final report.
|
||||
|
||||
To help with debugging builds a worker may send back some optional
|
||||
metadata:
|
||||
|
||||
**worker_name** (optional)
|
||||
The name of the worker.
|
||||
|
||||
**worker_hostname** (optional)
|
||||
The hostname of the worker.
|
||||
|
||||
**worker_ips** (optional)
|
||||
A list of IPs for the worker.
|
||||
|
||||
**worker_fqdn** (optional)
|
||||
The FQDN of the worker.
|
||||
|
||||
**worker_program** (optional)
|
||||
The program name of the worker. For example Jenkins or turbo-hipster.
|
||||
|
||||
**worker_version** (optional)
|
||||
The version of the software running the job.
|
||||
|
||||
**worker_extra** (optional)
|
||||
A dictionary of any extra metadata you may want to pass along.
|
||||
|
||||
It should then immediately send a WORK_STATUS packet with a value of 0
|
||||
percent complete. It may then optionally send subsequent WORK_STATUS
|
||||
packets with updated completion values.
|
||||
|
||||
When the build is complete, it should send a final WORK_DATA packet
|
||||
with the following in JSON format:
|
||||
|
||||
**result**
|
||||
Either the string 'SUCCESS' if the job succeeded, or any other value
|
||||
that describes the result if the job failed.
|
||||
|
||||
Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
|
||||
appropriate. A WORK_EXCEPTION packet will be interpreted as a
|
||||
WORK_FAIL, but the exception will be logged in Zuul's error log.
|
||||
|
||||
Stopping Builds
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
If Zuul needs to abort a build already in progress, it will invoke the
|
||||
following function through Gearman:
|
||||
|
||||
stop:MANAGER_NAME
|
||||
|
||||
Where **MANAGER_NAME** is the name of the manager worker supplied in
|
||||
the initial WORK_DATA packet when the job started. This is used to
|
||||
direct the stop: function invocation to the correct Gearman worker
|
||||
that is capable of stopping that particular job. The argument to the
|
||||
function should be the following encoded in JSON format:
|
||||
|
||||
**name**
|
||||
The job name of the build to stop.
|
||||
|
||||
**number**
|
||||
The build number of the build to stop.
|
||||
|
||||
The original job is expected to complete with a WORK_DATA and
|
||||
WORK_FAIL packet as described in `Starting Builds`_.
|
||||
|
||||
Build Descriptions
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to update the job running system with a description of the
|
||||
current state of all related builds, the job runner may optionally
|
||||
implement the following Gearman function:
|
||||
|
||||
set_description:MANAGER_NAME
|
||||
|
||||
Where **MANAGER_NAME** is used as described in `Stopping Builds`_.
|
||||
The argument to the function is the following encoded in JSON format:
|
||||
|
||||
**name**
|
||||
The job name of the build to describe.
|
||||
|
||||
**number**
|
||||
The build number of the build to describe.
|
||||
|
||||
**html_description**
|
||||
The description of the build in HTML format.
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
:title: Merger
|
||||
|
||||
Merger
|
||||
======
|
||||
|
||||
The Zuul Merger is a separate component which communicates with the
|
||||
main Zuul server via Gearman. Its purpose is to speculatively merge
|
||||
the changes for Zuul in preparation for testing. The resulting git
|
||||
commits also must be served to the test workers, and the server(s)
|
||||
running the Zuul Merger are expected to do this as well. Because both
|
||||
of these tasks are resource intensive, any number of Zuul Mergers can
|
||||
be run in parallel on distinct hosts.
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The Zuul Merger can read the same zuul.conf file as the main Zuul
|
||||
server and requires the ``gearman``, ``gerrit``, ``merger``, and
|
||||
``zuul`` sections (indicated fields only). Be sure the zuul_url is
|
||||
set appropriately on each host that runs a zuul-merger.
|
||||
|
||||
Zuul References
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
As the DependentPipelineManager may combine several changes together
|
||||
for testing when performing speculative execution, determining exactly
|
||||
how the workspace should be set up when running a Job can be complex.
|
||||
To alleviate this problem, Zuul performs merges itself, merging or
|
||||
cherry-picking changes as required and identifies the result with a
|
||||
Git reference of the form ``refs/zuul/<branch>/Z<random sha1>``.
|
||||
Preparing the workspace is then a simple matter of fetching that ref
|
||||
and checking it out. The parameters that provide this information are
|
||||
described in :ref:`launchers`.
|
||||
|
||||
These references need to be made available via a Git repository that
|
||||
is available to workers (such as Jenkins). This is accomplished by
|
||||
serving Zuul's Git repositories directly.
|
||||
|
||||
Serving Zuul Git Repos
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Zuul maintains its own copies of any needed Git repositories in the
|
||||
directory specified by ``git_dir`` in the ``merger`` section of
|
||||
zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
|
||||
Git repositories in order to provide Zuul refs for workers, you can
|
||||
configure Apache to do so using the following directives::
|
||||
|
||||
SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git
|
||||
SetEnv GIT_HTTP_EXPORT_ALL
|
||||
|
||||
AliasMatch ^/p/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/lib/zuul/git/$1
|
||||
AliasMatch ^/p/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/zuul/git/$1
|
||||
ScriptAlias /p/ /usr/lib/git-core/git-http-backend/
|
||||
|
||||
Note that Zuul's Git repositories are not bare, which means they have
|
||||
a working tree, and are not suitable for public consumption (for
|
||||
instance, a clone will produce a repository in an unpredictable state
|
||||
depending on what the state of Zuul's repository is when the clone
|
||||
happens). They are, however, suitable for automated systems that
|
||||
respond to Zuul triggers.
|
||||
|
||||
Clearing old references
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The references created under refs/zuul are not garbage collected. Since
|
||||
git fetch send them all to Gerrit to sync the repositories, the time
|
||||
spent on merge will slightly grow overtime and start being noticeable.
|
||||
|
||||
To clean them you can use the ``tools/zuul-clear-refs.py`` script on
|
||||
each repositories. It will delete Zuul references that point to commits
|
||||
for which the commit date is older than a given amount of days (default
|
||||
360)::
|
||||
|
||||
./tools/zuul-clear-refs.py /path/to/zuul/git/repo
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
Quick Start Guide
|
||||
=================
|
||||
|
||||
System Requirements
|
||||
-------------------
|
||||
|
||||
For most deployments zuul only needs 1-2GB. OpenStack uses a 30GB setup.
|
||||
|
||||
Install Zuul
|
||||
------------
|
||||
|
||||
You can get zuul from pypi via::
|
||||
|
||||
pip install zuul
|
||||
|
||||
Zuul Components
|
||||
---------------
|
||||
|
||||
Zuul provides the following components:
|
||||
|
||||
- **zuul-server**: scheduler daemon which communicates with Gerrit and
|
||||
Gearman. Handles receiving events, launching jobs, collecting results
|
||||
and postingreports.
|
||||
- **zuul-merger**: speculative-merger which communicates with Gearman.
|
||||
Prepares Git repositories for jobs to test against. This additionally
|
||||
requires a web server hosting the Git repositories which can be cloned
|
||||
by the jobs.
|
||||
- **zuul-cloner**: client side script used to setup job workspace. It is
|
||||
used to clone the repositories prepared by the zuul-merger described
|
||||
previously.
|
||||
- **gearmand**: optional builtin gearman daemon provided by zuul-server
|
||||
|
||||
External components:
|
||||
|
||||
- Jenkins Gearman plugin: Used by Jenkins to connect to Gearman
|
||||
|
||||
Zuul Communication
|
||||
------------------
|
||||
|
||||
All the Zuul components communicate with each other using Gearman. As well as
|
||||
the following communication channels:
|
||||
|
||||
zuul-server:
|
||||
|
||||
- Gerrit
|
||||
- Gearman Daemon
|
||||
|
||||
zuul-merger:
|
||||
|
||||
- Gerrit
|
||||
- Gearman Daemon
|
||||
|
||||
zuul-cloner:
|
||||
|
||||
- http hosted zuul-merger git repos
|
||||
|
||||
Jenkins:
|
||||
|
||||
- Gearman Daemon via Jenkins Gearman Plugin
|
||||
|
||||
Zuul Setup
|
||||
----------
|
||||
|
||||
At minimum we need to provide **zuul.conf** and **layout.yaml** and placed
|
||||
in /etc/zuul/ directory. You will also need a zuul user and ssh key for the
|
||||
zuul user in Gerrit. The following example uses the builtin gearmand service
|
||||
in zuul.
|
||||
|
||||
**zuul.conf**::
|
||||
|
||||
[zuul]
|
||||
layout_config=/etc/zuul/layout.yaml
|
||||
|
||||
[merger]
|
||||
git_dir=/git
|
||||
zuul_url=http://zuul.example.com/p
|
||||
|
||||
[gearman_server]
|
||||
start=true
|
||||
|
||||
[gearman]
|
||||
server=127.0.0.1
|
||||
|
||||
[connection gerrit]
|
||||
driver=gerrit
|
||||
server=git.example.com
|
||||
port=29418
|
||||
baseurl=https://git.example.com/gerrit/
|
||||
user=zuul
|
||||
sshkey=/home/zuul/.ssh/id_rsa
|
||||
|
||||
See :doc:`zuul` for more details.
|
||||
|
||||
The following sets up a basic timer triggered job using zuul.
|
||||
|
||||
**layout.yaml**::
|
||||
|
||||
pipelines:
|
||||
- name: periodic
|
||||
source: gerrit
|
||||
manager: IndependentPipelineManager
|
||||
trigger:
|
||||
timer:
|
||||
- time: '0 * * * *'
|
||||
|
||||
projects:
|
||||
- name: aproject
|
||||
periodic:
|
||||
- aproject-periodic-build
|
||||
|
||||
Starting Zuul
|
||||
-------------
|
||||
|
||||
You can run zuul-server with the **-d** option to make it not daemonize. It's
|
||||
a good idea at first to confirm there's no issues with your configuration.
|
||||
|
||||
Simply run::
|
||||
|
||||
zuul-server
|
||||
|
||||
Once run you should have 2 zuul-server processes::
|
||||
|
||||
zuul 12102 1 0 Jan21 ? 00:15:45 /home/zuul/zuulvenv/bin/python /home/zuul/zuulvenv/bin/zuul-server -d
|
||||
zuul 12107 12102 0 Jan21 ? 00:00:01 /home/zuul/zuulvenv/bin/python /home/zuul/zuulvenv/bin/zuul-server -d
|
||||
|
||||
Note: In this example zuul was installed in a virtualenv.
|
||||
|
||||
The 2nd zuul-server process is gearmand running if you are using the builtin
|
||||
gearmand server, otherwise there will only be 1 process.
|
||||
|
||||
Zuul won't actually process your Job queue however unless you also have a
|
||||
zuul-merger process running.
|
||||
|
||||
Simply run::
|
||||
|
||||
zuul-merger
|
||||
|
||||
Zuul should now be able to process your periodic job as configured above once
|
||||
the Jenkins side of things is configured.
|
||||
|
||||
Jenkins Setup
|
||||
-------------
|
||||
|
||||
Install the Jenkins Gearman Plugin via Jenkins Plugin management interface.
|
||||
Then naviage to **Manage > Configuration > Gearman** and setup the Jenkins
|
||||
server hostname/ip and port to connect to gearman.
|
||||
|
||||
At this point gearman should be running your Jenkins jobs.
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
Checking Gearman function registration (jobs). You can use telnet to connect
|
||||
to gearman to check that Jenkins is registering your configured jobs in
|
||||
gearman::
|
||||
|
||||
telnet <gearman_ip> 4730
|
||||
|
||||
Useful commands are **workers** and **status** which you can run by just
|
||||
typing those commands once connected to gearman. Every job in your Jenkins
|
||||
master must appear when you run **workers** for Zuul to be able to run jobs
|
||||
against your Jenkins instance.
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
:title: Reporters
|
||||
|
||||
Reporters
|
||||
=========
|
||||
|
||||
Zuul can communicate results and progress back to configurable
|
||||
protocols. For example, after succeeding in a build a pipeline can be
|
||||
configured to post a positive review back to Gerrit.
|
||||
|
||||
There are three stages when a report can be handled. That is on:
|
||||
Start, Success or Failure. Each stage can have multiple reports.
|
||||
For example, you can set verified on Gerrit and send an email.
|
||||
|
||||
Gerrit
|
||||
------
|
||||
|
||||
Zuul works with standard versions of Gerrit by invoking the
|
||||
``gerrit`` command over an SSH connection. It reports back to
|
||||
Gerrit using SSH.
|
||||
|
||||
The dictionary passed to the Gerrit reporter is used for ``gerrit
|
||||
review`` arguments, with the boolean value of ``true`` simply
|
||||
indicating that the argument should be present without following it
|
||||
with a value. For example, ``verified: 1`` becomes ``gerrit review
|
||||
--verified 1`` and ``submit: true`` becomes ``gerrit review
|
||||
--submit``.
|
||||
|
||||
A :ref:`connection` that uses the gerrit driver must be supplied to the
|
||||
trigger.
|
||||
|
||||
SMTP
|
||||
----
|
||||
|
||||
A simple email reporter is also available.
|
||||
|
||||
A :ref:`connection` that uses the smtp driver must be supplied to the
|
||||
reporter.
|
||||
|
||||
SMTP Configuration
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
zuul.conf contains the SMTP server and default to/from as described
|
||||
in :ref:`zuulconf`.
|
||||
|
||||
Each pipeline can overwrite the ``subject`` or the ``to`` or ``from`` address by
|
||||
providing alternatives as arguments to the reporter. For example, ::
|
||||
|
||||
pipelines:
|
||||
- name: post-merge
|
||||
manager: IndependentPipelineManager
|
||||
source: my_gerrit
|
||||
trigger:
|
||||
my_gerrit:
|
||||
- event: change-merged
|
||||
success:
|
||||
outgoing_smtp:
|
||||
to: you@example.com
|
||||
failure:
|
||||
internal_smtp:
|
||||
to: you@example.com
|
||||
from: alternative@example.com
|
||||
subject: Change {change} failed
|
||||
|
||||
SQL
|
||||
---
|
||||
|
||||
This reporter is used to store results in a database.
|
||||
|
||||
A :ref:`connection` that uses the sql driver must be supplied to the
|
||||
reporter.
|
||||
|
||||
SQL Configuration
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
zuul.conf contains the database connection and credentials. To store different
|
||||
reports in different databases you'll need to create a new connection per
|
||||
database.
|
||||
|
||||
The sql reporter is used to store the results from individual builds rather
|
||||
than the change. As such the sql reporter does nothing on "start" or
|
||||
"merge-failure".
|
||||
|
||||
**score**
|
||||
A score to store for the result of the build. eg: -1 might indicate a failed
|
||||
build similar to the vote posted back via the gerrit reporter.
|
||||
|
||||
For example ::
|
||||
|
||||
pipelines:
|
||||
- name: post-merge
|
||||
manager: IndependentPipelineManager
|
||||
source: my_gerrit
|
||||
trigger:
|
||||
my_gerrit:
|
||||
- event: change-merged
|
||||
success:
|
||||
mydb_conn:
|
||||
score: 1
|
||||
failure:
|
||||
mydb_conn:
|
||||
score: -1
|
||||
|
|
@ -1,104 +0,0 @@
|
|||
:title: Statsd reporting
|
||||
|
||||
Statsd reporting
|
||||
================
|
||||
|
||||
Zuul comes with support for the statsd protocol, when enabled and configured
|
||||
(see below), the Zuul scheduler will emit raw metrics to a statsd receiver
|
||||
which let you in turn generate nice graphics. An example is OpenStack Zuul
|
||||
status page: http://status.openstack.org/zuul/
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Statsd support uses the statsd python module. Note that Zuul will start without
|
||||
the statsd python module, so an existing Zuul installation may be missing it.
|
||||
|
||||
The configuration is done via environment variables STATSD_HOST and
|
||||
STATSD_PORT. They are interpreted by the statsd module directly and there is no
|
||||
such parameter in zuul.conf yet. Your init script will have to initialize both
|
||||
of them before launching Zuul.
|
||||
|
||||
Your init script most probably loads a configuration file named
|
||||
``/etc/default/zuul`` which would contain the environment variables::
|
||||
|
||||
$ cat /etc/default/zuul
|
||||
STATSD_HOST=10.0.0.1
|
||||
STATSD_PORT=8125
|
||||
|
||||
Metrics
|
||||
-------
|
||||
|
||||
The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`):
|
||||
|
||||
**gerrit.event.<type> (counters)**
|
||||
Gerrit emits different kind of message over its `stream-events` interface. As
|
||||
a convenience, Zuul emits metrics to statsd which save you from having to use
|
||||
a different daemon to measure Gerrit events.
|
||||
The Gerrit events have different types defined by Gerrit itself, Zuul will
|
||||
relay any type of event reusing the name defined by Gerrit. Some of the
|
||||
events emitted are:
|
||||
|
||||
* patchset-created
|
||||
* draft-published
|
||||
* change-abandonned
|
||||
* change-restored
|
||||
* change-merged
|
||||
* merge-failed
|
||||
* comment-added
|
||||
* ref-updated
|
||||
* reviewer-added
|
||||
|
||||
Refer to your Gerrit installation documentation for an exhaustive list of
|
||||
Gerrit event types.
|
||||
|
||||
**zuul.node_type.**
|
||||
Holds metrics specifc to build nodes per label. The hierarchy is:
|
||||
|
||||
#. **<build node label>** each of the labels associated to a build in
|
||||
Jenkins. It contains:
|
||||
|
||||
#. **job.<jobname>** subtree detailing per job statistics:
|
||||
|
||||
#. **wait_time** counter and timer of the wait time, with the
|
||||
difference of the job start time and the launch time, in
|
||||
milliseconds.
|
||||
|
||||
**zuul.pipeline.**
|
||||
Holds metrics specific to jobs. The hierarchy is:
|
||||
|
||||
#. **<pipeline name>** as defined in your `layout.yaml` file (ex: `gate`,
|
||||
`test`, `publish`). It contains:
|
||||
|
||||
#. **all_jobs** counter of jobs triggered by the pipeline.
|
||||
#. **current_changes** A gauge for the number of Gerrit changes being
|
||||
processed by this pipeline.
|
||||
#. **job** subtree detailing per jobs statistics:
|
||||
|
||||
#. **<jobname>** The triggered job name.
|
||||
#. **<build result>** Result as defined in your triggering system. For
|
||||
Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The
|
||||
metrics holds both an increasing counter and a timing
|
||||
reporting the duration of the build. Whenever the result is a
|
||||
SUCCESS or FAILURE, Zuul will additionally report the duration
|
||||
of the build as a timing event.
|
||||
|
||||
#. **resident_time** timing representing how long the Change has been
|
||||
known by Zuul (which includes build time and Zuul overhead).
|
||||
#. **total_changes** counter of the number of change proceeding since
|
||||
Zuul started.
|
||||
#. **wait_time** counter and timer of the wait time, with the difference
|
||||
of the job start time and the launch time, in milliseconds.
|
||||
|
||||
Additionally, the `zuul.pipeline.<pipeline name>` hierarchy contains
|
||||
`current_changes` (gauge), `resident_time` (timing) and `total_changes`
|
||||
(counter) metrics for each projects. The slash separator used in Gerrit name
|
||||
being replaced by dots.
|
||||
|
||||
As an example, given a job named `myjob` triggered by the `gate` pipeline
|
||||
which took 40 seconds to build, the Zuul scheduler will emit the following
|
||||
statsd events:
|
||||
|
||||
* `zuul.pipeline.gate.job.myjob.SUCCESS` +1
|
||||
* `zuul.pipeline.gate.job.myjob` 40 seconds
|
||||
* `zuul.pipeline.gate.all_jobs` +1
|
||||
|
|
@ -1,157 +0,0 @@
|
|||
:title: Triggers
|
||||
|
||||
Triggers
|
||||
========
|
||||
|
||||
The process of merging a change starts with proposing a change to be
|
||||
merged. Primarily, Zuul supports Gerrit as a triggering system.
|
||||
Zuul's design is modular, so alternate triggering and reporting
|
||||
systems can be supported.
|
||||
|
||||
Gerrit
|
||||
------
|
||||
|
||||
Zuul works with standard versions of Gerrit by invoking the ``gerrit
|
||||
stream-events`` command over an SSH connection. It also reports back
|
||||
to Gerrit using SSH.
|
||||
|
||||
If using Gerrit 2.7 or later, make sure the user is a member of a group
|
||||
that is granted the ``Stream Events`` permission, otherwise it will not
|
||||
be able to invoke the ``gerrit stream-events`` command over SSH.
|
||||
|
||||
A connection name with the gerrit driver can take multiple events with
|
||||
the following options.
|
||||
|
||||
**event**
|
||||
The event name from gerrit. Examples: ``patchset-created``,
|
||||
``comment-added``, ``ref-updated``. This field is treated as a
|
||||
regular expression.
|
||||
|
||||
**branch**
|
||||
The branch associated with the event. Example: ``master``. This
|
||||
field is treated as a regular expression, and multiple branches may
|
||||
be listed.
|
||||
|
||||
**ref**
|
||||
On ref-updated events, the branch parameter is not used, instead the
|
||||
ref is provided. Currently Gerrit has the somewhat idiosyncratic
|
||||
behavior of specifying bare refs for branch names (e.g., ``master``),
|
||||
but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
|
||||
Zuul matches what you put here exactly against what Gerrit
|
||||
provides. This field is treated as a regular expression, and
|
||||
multiple refs may be listed.
|
||||
|
||||
**ignore-deletes**
|
||||
When a branch is deleted, a ref-updated event is emitted with a newrev
|
||||
of all zeros specified. The ``ignore-deletes`` field is a boolean value
|
||||
that describes whether or not these newrevs trigger ref-updated events.
|
||||
The default is True, which will not trigger ref-updated events.
|
||||
|
||||
**approval**
|
||||
This is only used for ``comment-added`` events. It only matches if
|
||||
the event has a matching approval associated with it. Example:
|
||||
``code-review: 2`` matches a ``+2`` vote on the code review category.
|
||||
Multiple approvals may be listed.
|
||||
|
||||
**email**
|
||||
This is used for any event. It takes a regex applied on the performer
|
||||
email, i.e. Gerrit account email address. If you want to specify
|
||||
several email filters, you must use a YAML list. Make sure to use non
|
||||
greedy matchers and to escapes dots!
|
||||
Example: ``email: ^.*?@example\.org$``.
|
||||
|
||||
**email_filter** (deprecated)
|
||||
A deprecated alternate spelling of *email*. Only one of *email* or
|
||||
*email_filter* should be used.
|
||||
|
||||
**username**
|
||||
This is used for any event. It takes a regex applied on the performer
|
||||
username, i.e. Gerrit account name. If you want to specify several
|
||||
username filters, you must use a YAML list. Make sure to use non greedy
|
||||
matchers and to escapes dots!
|
||||
Example: ``username: ^jenkins$``.
|
||||
|
||||
**username_filter** (deprecated)
|
||||
A deprecated alternate spelling of *username*. Only one of *username* or
|
||||
*username_filter* should be used.
|
||||
|
||||
**comment**
|
||||
This is only used for ``comment-added`` events. It accepts a list of
|
||||
regexes that are searched for in the comment string. If any of these
|
||||
regexes matches a portion of the comment string the trigger is
|
||||
matched. ``comment: retrigger`` will match when comments
|
||||
containing 'retrigger' somewhere in the comment text are added to a
|
||||
change.
|
||||
|
||||
**comment_filter** (deprecated)
|
||||
A deprecated alternate spelling of *comment*. Only one of *comment* or
|
||||
*comment_filter* should be used.
|
||||
|
||||
*require-approval*
|
||||
This may be used for any event. It requires that a certain kind
|
||||
of approval be present for the current patchset of the change (the
|
||||
approval could be added by the event in question). It follows the
|
||||
same syntax as the :ref:`"approval" pipeline requirement
|
||||
<pipeline-require-approval>`. For each specified criteria there must
|
||||
exist a matching approval.
|
||||
|
||||
*reject-approval*
|
||||
This takes a list of approvals in the same format as
|
||||
*require-approval* but will fail to enter the pipeline if there is
|
||||
a matching approval.
|
||||
|
||||
|
||||
Timer
|
||||
-----
|
||||
|
||||
A simple timer trigger is available as well. It supports triggering
|
||||
jobs in a pipeline based on cron-style time instructions.
|
||||
|
||||
Timers don't require a special connection or driver. Instead they can
|
||||
be used by listing **timer** as the trigger.
|
||||
|
||||
This trigger will run based on a cron-style time specification.
|
||||
It will enqueue an event into its pipeline for every project
|
||||
defined in the configuration. Any job associated with the
|
||||
pipeline will run in response to that event.
|
||||
|
||||
**time**
|
||||
The time specification in cron syntax. Only the 5 part syntax is
|
||||
supported, not the symbolic names. Example: ``0 0 * * *`` runs
|
||||
at midnight.
|
||||
|
||||
Zuul
|
||||
----
|
||||
|
||||
The Zuul trigger generates events based on internal actions in Zuul.
|
||||
Multiple events may be listed.
|
||||
|
||||
Zuul events don't require a special connection or driver. Instead they
|
||||
can be used by listing **zuul** as the trigger.
|
||||
|
||||
**event**
|
||||
The event name. Currently supported:
|
||||
|
||||
*project-change-merged* when Zuul merges a change to a project,
|
||||
it generates this event for every open change in the project.
|
||||
|
||||
*parent-change-enqueued* when Zuul enqueues a change into any
|
||||
pipeline, it generates this event for every child of that
|
||||
change.
|
||||
|
||||
**pipeline**
|
||||
Only available for ``parent-change-enqueued`` events. This is the
|
||||
name of the pipeline in which the parent change was enqueued.
|
||||
|
||||
*require-approval*
|
||||
This may be used for any event. It requires that a certain kind
|
||||
of approval be present for the current patchset of the change (the
|
||||
approval could be added by the event in question). It follows the
|
||||
same syntax as the :ref:`"approval" pipeline requirement
|
||||
<pipeline-require-approval>`. For each specified criteria there must
|
||||
exist a matching approval.
|
||||
|
||||
*reject-approval*
|
||||
This takes a list of approvals in the same format as
|
||||
*require-approval* but will fail to enter the pipeline if there is
|
||||
a matching approval.
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
:title: Zuul Concepts
|
||||
|
||||
Zuul Concepts
|
||||
=============
|
||||
|
||||
Zuul's configuration is organized around the concept of a *pipeline*.
|
||||
In Zuul, a pipeline encompasses a workflow process which can be
|
||||
applied to one or more projects. For instance, a "check" pipeline
|
||||
might describe the actions which should cause newly proposed changes
|
||||
to projects to be tested. A "gate" pipeline might implement
|
||||
:ref:`project_gating` to automate merging changes to projects only if
|
||||
their tests pass. A "post" pipeline might update published
|
||||
documentation for a project when changes land.
|
||||
|
||||
The names "check", "gate", and "post" are arbitrary -- these are not
|
||||
concepts built into Zuul, but rather they are just a few common
|
||||
examples of workflow processes that can be described to Zuul and
|
||||
implemented as pipelines.
|
||||
|
||||
Once a pipeline has been defined, any number of projects may be
|
||||
associated with it, each one specifying what jobs should be run for
|
||||
that project in a given pipeline.
|
||||
|
||||
Pipelines have associated *triggers* which are descriptions of events
|
||||
which should cause something to be enqueued into a pipeline. For
|
||||
example, when a patchset is uploaded to Gerrit, a Gerrit
|
||||
*patchset-created* event is emitted. A pipeline configured to trigger
|
||||
on *patchset-created* events would then enqueue the associated change
|
||||
when Zuul receives that event. If there are jobs associated with that
|
||||
project and pipeline, they will be run. In addition to Gerrit, other
|
||||
triggers are available, including GitHub, timer, and zuul. See
|
||||
:ref:`drivers` for a full list of available triggers.
|
||||
|
||||
Once all of the jobs for an item in a pipeline have been run, the
|
||||
pipeline's *reporters* are responsible for reporting the results of
|
||||
all of the jobs. Continuing the example from earlier, if a pipeline
|
||||
were configured with a Gerrit reporter, it would leave a review
|
||||
comment on the change and set any approval votes that are configured.
|
||||
There are several reporting phases available; each of which may be
|
||||
configured with any number of reporters. See :ref:`drivers` for a
|
||||
full list of available reporters.
|
||||
|
||||
The items enqueued into a pipeline are each associated with a
|
||||
`git ref <https://git-scm.com/book/en/v2/Git-Internals-Git-References>`_.
|
||||
That ref may point to a proposed change, or it may be the tip of a
|
||||
branch or a tag. The triggering event determines the ref, and whether
|
||||
it represents a proposed or merged commit. Zuul prepares the ref for
|
||||
an item before running jobs. In the case of a proposed change, that
|
||||
means speculatively merging the change into its target branch. This
|
||||
means that any jobs that operate on the change will run with the git
|
||||
repo in the state it will be in after the change merges (which may be
|
||||
substantially different than the git repo state of the change itself
|
||||
since the repo may have merged other changes since the change was
|
||||
originally authored). Items in a pipeline may depend on other items,
|
||||
and if they do, all of their dependent changes will be included in the
|
||||
git repo state that Zuul prepares. For more detail on this process,
|
||||
see :ref:`project_gating` and :ref:`dependencies`.
|
||||
|
||||
The configuration for nearly everything described above is held in
|
||||
files inside of the git repos upon which Zuul operates. Zuul's
|
||||
configuration is global, but distributed. Jobs which are defined in
|
||||
one project might be used in another project while pipelines are
|
||||
available to all projects. When Zuul starts, it reads its
|
||||
configuration from all of the projects it knows about, and when
|
||||
changes to its configuration are proposed, those changes may take
|
||||
effect temporarily as part of the proposed change, or immediately
|
||||
after the change merges, depending on the type of project in which the
|
||||
change appears.
|
||||
|
||||
Jobs specify the type and quantity of nodes which they require.
|
||||
Before executing each job, Zuul will contact its companion program,
|
||||
Nodepool, to supply them. Nodepool may be configured to supply static
|
||||
nodes or contact cloud providers to create or delete nodes as
|
||||
necessary. The types of nodes available to Zuul are determined by the
|
||||
Nodepool administrator.
|
||||
|
||||
The executable contents of jobs themselves are Ansible playbooks.
|
||||
Ansible's support for orchestrating tasks on remote nodes is
|
||||
particularly suited to Zuul's support for multi-node testing. Ansible
|
||||
is also easy to use for simple tasks (such as executing a shell
|
||||
script) or sophisticated deployment scenarios. When Zuul runs
|
||||
Ansible, it attempts to do so in a manner most similar to the way that
|
||||
Ansible might be used to orchestrate remote systems. Ansible itself
|
||||
is run on the :ref:`executor <executor>` and acts remotely upon the test
|
||||
nodes supplied to a job. This facilitates continuous delivery by making it
|
||||
possible to use the same Ansible playbooks in testing and production.
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,62 @@
|
|||
:title: Encryption
|
||||
|
||||
.. _encryption:
|
||||
|
||||
Encryption
|
||||
==========
|
||||
|
||||
Zuul supports storing encrypted data directly in the git repositories
|
||||
of projects it operates on. If you have a job which requires private
|
||||
information in order to run (e.g., credentials to interact with a
|
||||
third-party service) those credentials can be stored along with the
|
||||
job definition.
|
||||
|
||||
Each project in Zuul has its own automatically generated RSA keypair
|
||||
which can be used by anyone to encrypt a secret and only Zuul is able
|
||||
to decrypt it. Zuul serves each project's public key using its
|
||||
build-in webserver. They can be fetched at the path
|
||||
``/<tenant>/<project>.pub`` where ``<project>`` is the canonical name
|
||||
of a project and ``<tenant>`` is the name of a tenant with that project.
|
||||
|
||||
Zuul currently supports one encryption scheme, PKCS#1 with OAEP, which
|
||||
can not store secrets longer than the 3760 bits (derived from the key
|
||||
length of 4096 bits minus 336 bits of overhead). The padding used by
|
||||
this scheme ensures that someone examining the encrypted data can not
|
||||
determine the length of the plaintext version of the data, except to
|
||||
know that it is not longer than 3760 bits (or some multiple thereof).
|
||||
|
||||
In the config files themselves, Zuul uses an extensible method of
|
||||
specifying the encryption scheme used for a secret so that other
|
||||
schemes may be added later. To specify a secret, use the
|
||||
``!encrypted/pkcs1-oaep`` YAML tag along with the base64 encoded
|
||||
value. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- secret:
|
||||
name: test_secret
|
||||
data:
|
||||
password: !encrypted/pkcs1-oaep |
|
||||
BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi
|
||||
...
|
||||
|
||||
To support secrets longer than 3760 bits, the value after the
|
||||
encryption tag may be a list rather than a scalar. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- secret:
|
||||
name: long_secret
|
||||
data:
|
||||
password: !encrypted/pkcs1-oaep
|
||||
- er1UXNOD3OqtsRJaP0Wvaqiqx0ZY2zzRt6V9vqIsRaz1R5C4/AEtIad/DERZHwk3Nk+KV
|
||||
...
|
||||
- HdWDS9lCBaBJnhMsm/O9tpzCq+GKRELpRzUwVgU5k822uBwhZemeSrUOLQ8hQ7q/vVHln
|
||||
...
|
||||
|
||||
Zuul provides a standalone script to make encrypting values easy; it
|
||||
can be found at `tools/encrypt_secret.py` in the Zuul source
|
||||
directory.
|
||||
|
||||
.. program-output:: python3 ../../tools/encrypt_secret.py --help
|
||||
|
||||
|
|
@ -0,0 +1,398 @@
|
|||
:title: Project Gating
|
||||
|
||||
.. _project_gating:
|
||||
|
||||
Project Gating
|
||||
==============
|
||||
|
||||
Traditionally, many software development projects merge changes from
|
||||
developers into the repository, and then identify regressions
|
||||
resulting from those changes (perhaps by running a test suite with a
|
||||
continuous integration system), followed by more patches to fix those
|
||||
bugs. When the mainline of development is broken, it can be very
|
||||
frustrating for developers and can cause lost productivity,
|
||||
particularly so when the number of contributors or contributions is
|
||||
large.
|
||||
|
||||
The process of gating attempts to prevent changes that introduce
|
||||
regressions from being merged. This keeps the mainline of development
|
||||
open and working for all developers, and only when a change is
|
||||
confirmed to work without disruption is it merged.
|
||||
|
||||
Many projects practice an informal method of gating where developers
|
||||
with mainline commit access ensure that a test suite runs before
|
||||
merging a change. With more developers, more changes, and more
|
||||
comprehensive test suites, that process does not scale very well, and
|
||||
is not the best use of a developer's time. Zuul can help automate
|
||||
this process, with a particular emphasis on ensuring large numbers of
|
||||
changes are tested correctly.
|
||||
|
||||
Testing in parallel
|
||||
-------------------
|
||||
|
||||
A particular focus of Zuul is ensuring correctly ordered testing of
|
||||
changes in parallel. A gating system should always test each change
|
||||
applied to the tip of the branch exactly as it is going to be merged.
|
||||
A simple way to do that would be to test one change at a time, and
|
||||
merge it only if it passes tests. That works very well, but if
|
||||
changes take a long time to test, developers may have to wait a long
|
||||
time for their changes to make it into the repository. With some
|
||||
projects, it may take hours to test changes, and it is easy for
|
||||
developers to create changes at a rate faster than they can be tested
|
||||
and merged.
|
||||
|
||||
Zuul's :value:`dependent pipeline manager<pipeline.manager.dependent>`
|
||||
allows for parallel execution of test jobs for gating while ensuring
|
||||
changes are tested correctly, exactly as if they had been tested one
|
||||
at a time. It does this by performing speculative execution of test
|
||||
jobs; it assumes that all jobs will succeed and tests them in parallel
|
||||
accordingly. If they do succeed, they can all be merged. However, if
|
||||
one fails, then changes that were expecting it to succeed are
|
||||
re-tested without the failed change. In the best case, as many
|
||||
changes as execution contexts are available may be tested in parallel
|
||||
and merged at once. In the worst case, changes are tested one at a
|
||||
time (as each subsequent change fails, changes behind it start again).
|
||||
|
||||
For example, if a core developer approves five changes in rapid
|
||||
succession::
|
||||
|
||||
A, B, C, D, E
|
||||
|
||||
Zuul queues those changes in the order they were approved, and notes
|
||||
that each subsequent change depends on the one ahead of it merging:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
A <- B <- C <- D <- E;
|
||||
}
|
||||
|
||||
Zuul then starts immediately testing all of the changes in parallel.
|
||||
But in the case of changes that depend on others, it instructs the
|
||||
test system to include the changes ahead of it, with the assumption
|
||||
they pass. That means jobs testing change *B* include change *A* as
|
||||
well::
|
||||
|
||||
Jobs for A: merge change A, then test
|
||||
Jobs for B: merge changes A and B, then test
|
||||
Jobs for C: merge changes A, B and C, then test
|
||||
Jobs for D: merge changes A, B, C and D, then test
|
||||
Jobs for E: merge changes A, B, C, D and E, then test
|
||||
|
||||
Hence jobs triggered to tests A will only test A and ignore B, C, D:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
master -> A -> B -> C -> D -> E;
|
||||
group jobs_for_A {
|
||||
label = "Merged changes for A";
|
||||
master -> A;
|
||||
}
|
||||
group ignored_to_test_A {
|
||||
label = "Ignored changes";
|
||||
color = "lightgray";
|
||||
B -> C -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
The jobs for E would include the whole dependency chain: A, B, C, D, and E.
|
||||
E will be tested assuming A, B, C, and D passed:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
group jobs_for_E {
|
||||
label = "Merged changes for E";
|
||||
master -> A -> B -> C -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
If changes *A* and *B* pass tests (green), and *C*, *D*, and *E* fail (red):
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
A [color = lightgreen];
|
||||
B [color = lightgreen];
|
||||
C [color = pink];
|
||||
D [color = pink];
|
||||
E [color = pink];
|
||||
|
||||
master <- A <- B <- C <- D <- E;
|
||||
}
|
||||
|
||||
Zuul will merge change *A* followed by change *B*, leaving this queue:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
C [color = pink];
|
||||
D [color = pink];
|
||||
E [color = pink];
|
||||
|
||||
C <- D <- E;
|
||||
}
|
||||
|
||||
Since *D* was dependent on *C*, it is not clear whether *D*'s failure is the
|
||||
result of a defect in *D* or *C*:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
C [color = pink];
|
||||
D [label = "D\n?"];
|
||||
E [label = "E\n?"];
|
||||
|
||||
C <- D <- E;
|
||||
}
|
||||
|
||||
Since *C* failed, Zuul will report its failure and drop *C* from the queue,
|
||||
keeping D and E:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
|
||||
D [label = "D\n?"];
|
||||
E [label = "E\n?"];
|
||||
|
||||
D <- E;
|
||||
}
|
||||
|
||||
This queue is the same as if two new changes had just arrived, so Zuul
|
||||
starts the process again testing *D* against the tip of the branch, and
|
||||
*E* against *D*:
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
master -> D -> E;
|
||||
group jobs_for_D {
|
||||
label = "Merged changes for D";
|
||||
master -> D;
|
||||
}
|
||||
group ignored_to_test_D {
|
||||
label = "Skip";
|
||||
color = "lightgray";
|
||||
E;
|
||||
}
|
||||
}
|
||||
|
||||
.. blockdiag::
|
||||
|
||||
blockdiag foo {
|
||||
node_width = 40;
|
||||
span_width = 40;
|
||||
group jobs_for_E {
|
||||
label = "Merged changes for E";
|
||||
master -> D -> E;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Cross Project Testing
|
||||
---------------------
|
||||
|
||||
When your projects are closely coupled together, you want to make sure
|
||||
changes entering the gate are going to be tested with the version of
|
||||
other projects currently enqueued in the gate (since they will
|
||||
eventually be merged and might introduce breaking features).
|
||||
|
||||
Such relationships can be defined in Zuul configuration by placing
|
||||
projects in a shared queue within a dependent pipeline. Whenever
|
||||
changes for any project enter a pipeline with such a shared queue,
|
||||
they are tested together, such that the commits for the changes ahead
|
||||
in the queue are automatically present in the jobs for the changes
|
||||
behind them. See :ref:`project` for more details.
|
||||
|
||||
A given dependent pipeline may have as many shared change queues as
|
||||
necessary, so groups of related projects may share a change queue
|
||||
without interfering with unrelated projects.
|
||||
:value:`Independent pipelines <pipeline.manager.independent>` do
|
||||
not use shared change queues, however, they may still be used to test
|
||||
changes across projects using cross-project dependencies.
|
||||
|
||||
.. _dependencies:
|
||||
|
||||
Cross-Project Dependencies
|
||||
--------------------------
|
||||
|
||||
Zuul permits users to specify dependencies across projects. Using a
|
||||
special footer in Git commit messages, users may specify that a change
|
||||
depends on another change in any repository known to Zuul.
|
||||
|
||||
Zuul's cross-project dependencies behave like a directed acyclic graph
|
||||
(DAG), like git itself, to indicate a one-way dependency relationship
|
||||
between changes in different git repositories. Change A may depend on
|
||||
B, but B may not depend on A.
|
||||
|
||||
To use them, include ``Depends-On: <change-url>`` in the footer of a
|
||||
commit message. For example, a change which depends on a GitHub pull
|
||||
request (PR #4) might have the following footer::
|
||||
|
||||
Depends-On: https://github.com/example/test/pull/4
|
||||
|
||||
And a change which depends on a Gerrit change (change number 3)::
|
||||
|
||||
Depends-On: https://review.example.com/3
|
||||
|
||||
Changes may depend on changes in any other project, even projects not
|
||||
on the same system (i.e., a Gerrit change may depend on a GitHub pull
|
||||
request).
|
||||
|
||||
.. note::
|
||||
|
||||
An older syntax of specifying dependencies using Gerrit change-ids
|
||||
is still supported, however it is deprecated and will be removed in
|
||||
a future version.
|
||||
|
||||
Dependent Pipeline
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When Zuul sees changes with cross-project dependencies, it serializes
|
||||
them in the usual manner when enqueuing them into a pipeline. This
|
||||
means that if change A depends on B, then when they are added to a
|
||||
dependent pipeline, B will appear first and A will follow:
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crd {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class greendot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
color = green,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
A_status [ class = greendot ]
|
||||
B_status [ class = greendot ]
|
||||
B_status -- A_status
|
||||
|
||||
'Change B\nURL: .../4' <- 'Change A\nDepends-On: .../4'
|
||||
}
|
||||
|
||||
If tests for B fail, both B and A will be removed from the pipeline, and
|
||||
it will not be possible for A to merge until B does.
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
If changes with cross-project dependencies do not share a change
|
||||
queue then Zuul is unable to enqueue them together, and the first
|
||||
will be required to merge before the second is enqueued.
|
||||
|
||||
Independent Pipeline
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When changes are enqueued into an independent pipeline, all of the
|
||||
related dependencies (both normal git-dependencies that come from
|
||||
parent commits as well as cross-project dependencies) appear in a
|
||||
dependency graph, as in a dependent pipeline. This means that even in
|
||||
an independent pipeline, your change will be tested with its
|
||||
dependencies. Changes that were previously unable to be fully tested
|
||||
until a related change landed in a different repository may now be
|
||||
tested together from the start.
|
||||
|
||||
All of the changes are still independent (you will note that the whole
|
||||
pipeline does not share a graph as in a dependent pipeline), but for
|
||||
each change tested, all of its dependencies are visually connected to
|
||||
it, and they are used to construct the git repositories that Zuul uses
|
||||
when testing.
|
||||
|
||||
When looking at this graph on the status page, you will note that the
|
||||
dependencies show up as grey dots, while the actual change tested shows
|
||||
up as red or green (depending on the jobs results):
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crdgrey {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class dot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
A_status [class = "dot", color = green]
|
||||
B_status [class = "dot", color = grey]
|
||||
B_status -- A_status
|
||||
|
||||
"Change B\nURL: .../4" <- "Change A\nDepends-On: .../4"
|
||||
}
|
||||
|
||||
This is to indicate that the grey changes are only there to establish
|
||||
dependencies. Even if one of the dependencies is also being tested, it
|
||||
will show up as a grey dot when used as a dependency, but separately and
|
||||
additionally will appear as its own red or green dot for its test.
|
||||
|
||||
|
||||
Multiple Changes
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
A change may list more than one dependency by simply adding more
|
||||
``Depends-On:`` lines to the commit message footer. It is possible
|
||||
for a change in project A to depend on a change in project B and a
|
||||
change in project C.
|
||||
|
||||
.. blockdiag::
|
||||
:align: center
|
||||
|
||||
blockdiag crdmultichanges {
|
||||
orientation = portrait
|
||||
span_width = 30
|
||||
class greendot [
|
||||
label = "",
|
||||
shape = circle,
|
||||
color = green,
|
||||
width = 20, height = 20
|
||||
]
|
||||
|
||||
C_status [ class = "greendot" ]
|
||||
B_status [ class = "greendot" ]
|
||||
A_status [ class = "greendot" ]
|
||||
C_status -- B_status -- A_status
|
||||
|
||||
A [ label = "Repo A\nDepends-On: .../3\nDepends-On: .../4" ]
|
||||
group {
|
||||
orientation = portrait
|
||||
label = "Dependencies"
|
||||
color = "lightgray"
|
||||
|
||||
B [ label = "Repo B\nURL: .../3" ]
|
||||
C [ label = "Repo C\nURL: .../4" ]
|
||||
}
|
||||
B, C <- A
|
||||
}
|
||||
|
||||
Cycles
|
||||
~~~~~~
|
||||
|
||||
If a cycle is created by use of cross-project dependencies, Zuul will
|
||||
abort its work very early. There will be no message in Gerrit and no
|
||||
changes that are part of the cycle will be enqueued into any pipeline.
|
||||
This is to protect Zuul from infinite loops.
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
User's Guide
|
||||
============
|
||||
|
||||
This guide is for all users of Zuul. If you work on a project where
|
||||
Zuul is used to drive automation (whether that's testing proposed
|
||||
changes, building artifacts, or deploying builds), this guide will
|
||||
help you understand the concepts that underlie Zuul, and how to
|
||||
configure it to meet your needs.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
concepts
|
||||
gating
|
||||
config
|
||||
jobs
|
||||
encryption
|
||||
|
|
@ -0,0 +1,576 @@
|
|||
:title: Job Content
|
||||
|
||||
Job Content
|
||||
===========
|
||||
|
||||
Zuul jobs are implemented as Ansible playbooks. Zuul prepares the
|
||||
repositories used for a job, installs any required Ansible roles, and
|
||||
then executes the job's playbooks. Any setup or artifact collection
|
||||
required is the responsibility of the job itself. While this flexible
|
||||
arrangement allows for almost any kind of job to be run by Zuul,
|
||||
batteries are included. Zuul has a standard library of jobs upon
|
||||
which to build.
|
||||
|
||||
Working Directory
|
||||
-----------------
|
||||
|
||||
Before starting each job, the Zuul executor creates a directory to
|
||||
hold all of the content related to the job. This includes some
|
||||
directories which are used by Zuul to configure and run Ansible and
|
||||
may not be accessible, as well as a directory tree, under ``work/``,
|
||||
that is readable and writable by the job. The hierarchy is:
|
||||
|
||||
**work/**
|
||||
The working directory of the job.
|
||||
|
||||
**work/src/**
|
||||
Contains the prepared git repositories for the job.
|
||||
|
||||
**work/logs/**
|
||||
Where the Ansible log for the job is written; your job
|
||||
may place other logs here as well.
|
||||
|
||||
Git Repositories
|
||||
----------------
|
||||
|
||||
The git repositories in ``work/src`` contain the repositories for all
|
||||
of the projects specified in the ``required-projects`` section of the
|
||||
job, plus the project associated with the queue item if it isn't
|
||||
already in that list. In the case of a proposed change, that change
|
||||
and all of the changes ahead of it in the pipeline queue will already
|
||||
be merged into their respective repositories and target branches. The
|
||||
change's project will have the change's branch checked out, as will
|
||||
all of the other projects, if that branch exists (otherwise, a
|
||||
fallback or default branch will be used). If your job needs to
|
||||
operate on multiple branches, simply checkout the appropriate branches
|
||||
of these git repos to ensure that the job results reflect the proposed
|
||||
future state that Zuul is testing, and all dependencies are present.
|
||||
Do not use any git remotes; the local repositories are guaranteed to
|
||||
be up to date.
|
||||
|
||||
The repositories will be placed on the filesystem in directories
|
||||
corresponding with the canonical hostname of their source connection.
|
||||
For example::
|
||||
|
||||
work/src/git.example.com/project1
|
||||
work/src/github.com/project2
|
||||
|
||||
Is the layout that would be present for a job which included project1
|
||||
from the connection associated to git.example.com and project2 from
|
||||
GitHub. This helps avoid collisions between projects with the same
|
||||
name, and some language environments, such as Go, expect repositories
|
||||
in this format.
|
||||
|
||||
Note that these git repositories are located on the executor; in order
|
||||
to be useful to most kinds of jobs, they will need to be present on
|
||||
the test nodes. The ``base`` job in the standard library (see
|
||||
`zuul-base-jobs documentation`_ for details) contains a
|
||||
pre-playbook which copies the repositories to all of the job's nodes.
|
||||
It is recommended to always inherit from this base job to ensure that
|
||||
behavior.
|
||||
|
||||
.. _zuul-base-jobs documentation: https://docs.openstack.org/infra/zuul-base-jobs/jobs.html#job-base
|
||||
|
||||
.. TODO: document src (and logs?) directory
|
||||
|
||||
Variables
|
||||
---------
|
||||
|
||||
There are several sources of variables which are available to Ansible:
|
||||
variables defined in jobs, secrets, and site-wide variables. The
|
||||
order of precedence is:
|
||||
|
||||
* Site-wide variables
|
||||
|
||||
* Secrets
|
||||
|
||||
* Job variables
|
||||
|
||||
* Parent job results
|
||||
|
||||
Meaning that a site-wide variable with the same name as any other will
|
||||
override its value, and similarly, secrets override job variables of
|
||||
the same name which override data returned from parent jobs. Each of
|
||||
the sources is described below.
|
||||
|
||||
|
||||
Job Variables
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Any variables specified in the job definition (using the
|
||||
:attr:`job.vars` attribute) are available as Ansible host variables.
|
||||
They are added to the ``vars`` section of the inventory file under the
|
||||
``all`` hosts group, so they are available to all hosts. Simply refer
|
||||
to them by the name specified in the job's ``vars`` section.
|
||||
|
||||
Secrets
|
||||
~~~~~~~
|
||||
|
||||
:ref:`Secrets <secret>` also appear as variables available to Ansible.
|
||||
Unlike job variables, these are not added to the inventory file (so
|
||||
that the inventory file may be kept for debugging purposes without
|
||||
revealing secrets). But they are still available to Ansible as normal
|
||||
variables. Because secrets are groups of variables, they will appear
|
||||
as a dictionary structure in templates, with the dictionary itself
|
||||
being the name of the secret, and its members the individual items in
|
||||
the secret. For example, a secret defined as:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- secret:
|
||||
name: credentials
|
||||
data:
|
||||
username: foo
|
||||
password: bar
|
||||
|
||||
Might be used in a template as::
|
||||
|
||||
{{ credentials.username }} {{ credentials.password }}
|
||||
|
||||
Secrets are only available to playbooks associated with the job
|
||||
definition which uses the secret; they are not available to playbooks
|
||||
associated with child jobs or job variants.
|
||||
|
||||
Zuul Variables
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Zuul supplies not only the variables specified by the job definition
|
||||
to Ansible, but also some variables from Zuul itself.
|
||||
|
||||
When a pipeline is triggered by an action, it enqueues items which may
|
||||
vary based on the pipeline's configuration. For example, when a new
|
||||
change is created, that change may be enqueued into the pipeline,
|
||||
while a tag may be enqueued into the pipeline when it is pushed.
|
||||
|
||||
Information about these items is available to jobs. All of the items
|
||||
enqueued in a pipeline are git references, and therefore share some
|
||||
attributes in common. But other attributes may vary based on the type
|
||||
of item.
|
||||
|
||||
.. var:: zuul
|
||||
|
||||
All items provide the following information as Ansible variables
|
||||
under the ``zuul`` key:
|
||||
|
||||
.. var:: build
|
||||
|
||||
The UUID of the build. A build is a single execution of a job.
|
||||
When an item is enqueued into a pipeline, this usually results
|
||||
in one build of each job configured for that item's project.
|
||||
However, items may be re-enqueued in which case another build
|
||||
may run. In dependent pipelines, the same job may run multiple
|
||||
times for the same item as circumstances change ahead in the
|
||||
queue. Each time a job is run, for whatever reason, it is
|
||||
acompanied with a new unique id.
|
||||
|
||||
.. var:: buildset
|
||||
|
||||
The build set UUID. When Zuul runs jobs for an item, the
|
||||
collection of those jobs is known as a buildset. If the
|
||||
configuration of items ahead in a dependent pipeline changes,
|
||||
Zuul creates a new buildset and restarts all of the jobs.
|
||||
|
||||
.. var:: ref
|
||||
|
||||
The git ref of the item. This will be the full path (e.g.,
|
||||
`refs/heads/master` or `refs/changes/...`).
|
||||
|
||||
.. var:: override_checkout
|
||||
|
||||
If the job was configured to override the branch or tag checked
|
||||
out, this will contain the specified value. Otherwise, this
|
||||
variable will be undefined.
|
||||
|
||||
.. var:: pipeline
|
||||
|
||||
The name of the pipeline in which the job is being run.
|
||||
|
||||
.. var:: job
|
||||
|
||||
The name of the job being run.
|
||||
|
||||
.. var:: voting
|
||||
|
||||
A boolean indicating whether the job is voting.
|
||||
|
||||
.. var:: project
|
||||
|
||||
The item's project. This is a data structure with the following
|
||||
fields:
|
||||
|
||||
.. var:: name
|
||||
|
||||
The name of the project, excluding hostname. E.g., `org/project`.
|
||||
|
||||
.. var:: short_name
|
||||
|
||||
The name of the project, excluding directories or
|
||||
organizations. E.g., `project`.
|
||||
|
||||
.. var:: canonical_hostname
|
||||
|
||||
The canonical hostname where the project lives. E.g.,
|
||||
`git.example.com`.
|
||||
|
||||
.. var:: canonical_name
|
||||
|
||||
The full canonical name of the project including hostname.
|
||||
E.g., `git.example.com/org/project`.
|
||||
|
||||
.. var:: src_dir
|
||||
|
||||
The path to the source code relative to the work dir. E.g.,
|
||||
`src/git.example.com/org/project`.
|
||||
|
||||
.. var:: projects
|
||||
:type: dict
|
||||
|
||||
A dictionary of all projects prepared by Zuul for the item. It
|
||||
includes, at least, the item's own project. It also includes
|
||||
the projects of any items this item depends on, as well as the
|
||||
projects that appear in :attr:`job.required-projects`.
|
||||
|
||||
This is a dictionary of dictionaries. Each value has a key of
|
||||
the `canonical_name`, then each entry consists of:
|
||||
|
||||
.. var:: name
|
||||
|
||||
The name of the project, excluding hostname. E.g., `org/project`.
|
||||
|
||||
.. var:: short_name
|
||||
|
||||
The name of the project, excluding directories or
|
||||
organizations. E.g., `project`.
|
||||
|
||||
.. var:: canonical_hostname
|
||||
|
||||
The canonical hostname where the project lives. E.g.,
|
||||
`git.example.com`.
|
||||
|
||||
.. var:: canonical_name
|
||||
|
||||
The full canonical name of the project including hostname.
|
||||
E.g., `git.example.com/org/project`.
|
||||
|
||||
.. var:: src_dir
|
||||
|
||||
The path to the source code, relative to the work dir. E.g.,
|
||||
`src/git.example.com/org/project`.
|
||||
|
||||
.. var:: required
|
||||
|
||||
A boolean indicating whether this project appears in the
|
||||
:attr:`job.required-projects` list for this job.
|
||||
|
||||
.. var:: checkout
|
||||
|
||||
The branch or tag that Zuul checked out for this project.
|
||||
This may be influenced by the branch or tag associated with
|
||||
the item as well as the job configuration.
|
||||
|
||||
For example, to access the source directory of a single known
|
||||
project, you might use::
|
||||
|
||||
{{ zuul.projects['git.example.com/org/project'].src_dir }}
|
||||
|
||||
To iterate over the project list, you might write a task
|
||||
something like::
|
||||
|
||||
- name: Sample project iteration
|
||||
debug:
|
||||
msg: "Project {{ item.name }} is at {{ item.src_dir }}
|
||||
with_items: {{ zuul.projects.values() | list }}
|
||||
|
||||
|
||||
.. var:: _projects
|
||||
:type: dict
|
||||
|
||||
The same as ``projects`` but a dictionary indexed by the
|
||||
``name`` value of each entry. ``projects`` will be converted to
|
||||
this.
|
||||
|
||||
.. var:: tenant
|
||||
|
||||
The name of the current Zuul tenant.
|
||||
|
||||
.. var:: timeout
|
||||
|
||||
The job timeout, in seconds.
|
||||
|
||||
.. var:: jobtags
|
||||
|
||||
A list of tags associated with the job. Not to be confused with
|
||||
git tags, these are simply free-form text fields that can be
|
||||
used by the job for reporting or classification purposes.
|
||||
|
||||
.. var:: items
|
||||
:type: list
|
||||
|
||||
A list of dictionaries, each representing an item being tested
|
||||
with this change with the format:
|
||||
|
||||
.. var:: project
|
||||
|
||||
The item's project. This is a data structure with the
|
||||
following fields:
|
||||
|
||||
.. var:: name
|
||||
|
||||
The name of the project, excluding hostname. E.g.,
|
||||
`org/project`.
|
||||
|
||||
.. var:: short_name
|
||||
|
||||
The name of the project, excluding directories or
|
||||
organizations. E.g., `project`.
|
||||
|
||||
.. var:: canonical_hostname
|
||||
|
||||
The canonical hostname where the project lives. E.g.,
|
||||
`git.example.com`.
|
||||
|
||||
.. var:: canonical_name
|
||||
|
||||
The full canonical name of the project including hostname.
|
||||
E.g., `git.example.com/org/project`.
|
||||
|
||||
.. var:: src_dir
|
||||
|
||||
The path to the source code on the remote host, relative
|
||||
to the home dir of the remote user.
|
||||
E.g., `src/git.example.com/org/project`.
|
||||
|
||||
.. var:: branch
|
||||
|
||||
The target branch of the change (without the `refs/heads/` prefix).
|
||||
|
||||
.. var:: change
|
||||
|
||||
The identifier for the change.
|
||||
|
||||
.. var:: change_url
|
||||
|
||||
The URL to the source location of the given change.
|
||||
E.g., `https://review.example.org/#/c/123456/` or
|
||||
`https://github.com/example/example/pull/1234`.
|
||||
|
||||
.. var:: patchset
|
||||
|
||||
The patchset identifier for the change. If a change is
|
||||
revised, this will have a different value.
|
||||
|
||||
.. var:: zuul_success
|
||||
|
||||
Post run playbook(s) will be passed this variable to indicate if the run
|
||||
phase of the job was successful or not. This variable is meant to be used
|
||||
with the `boolean` filter.
|
||||
|
||||
|
||||
Change Items
|
||||
++++++++++++
|
||||
|
||||
A change to the repository. Most often, this will be a git reference
|
||||
which has not yet been merged into the repository (e.g., a gerrit
|
||||
change or a GitHub pull request). The following additional variables
|
||||
are available:
|
||||
|
||||
.. var:: zuul
|
||||
:hidden:
|
||||
|
||||
.. var:: branch
|
||||
|
||||
The target branch of the change (without the `refs/heads/` prefix).
|
||||
|
||||
.. var:: change
|
||||
|
||||
The identifier for the change.
|
||||
|
||||
.. var:: patchset
|
||||
|
||||
The patchset identifier for the change. If a change is revised,
|
||||
this will have a different value.
|
||||
|
||||
.. var:: change_url
|
||||
|
||||
The URL to the source location of the given change.
|
||||
E.g., `https://review.example.org/#/c/123456/` or
|
||||
`https://github.com/example/example/pull/1234`.
|
||||
|
||||
Branch Items
|
||||
++++++++++++
|
||||
|
||||
This represents a branch tip. This item may have been enqueued
|
||||
because the branch was updated (via a change having merged, or a
|
||||
direct push). Or it may have been enqueued by a timer for the purpose
|
||||
of verifying the current condition of the branch. The following
|
||||
additional variables are available:
|
||||
|
||||
.. var:: zuul
|
||||
:hidden:
|
||||
|
||||
.. var:: branch
|
||||
|
||||
The name of the item's branch (without the `refs/heads/`
|
||||
prefix).
|
||||
|
||||
.. var:: oldrev
|
||||
|
||||
If the item was enqueued as the result of a change merging or
|
||||
being pushed to the branch, the git sha of the old revision will
|
||||
be included here. Otherwise, this variable will be undefined.
|
||||
|
||||
.. var:: newrev
|
||||
|
||||
If the item was enqueued as the result of a change merging or
|
||||
being pushed to the branch, the git sha of the new revision will
|
||||
be included here. Otherwise, this variable will be undefined.
|
||||
|
||||
Tag Items
|
||||
+++++++++
|
||||
|
||||
This represents a git tag. The item may have been enqueued because a
|
||||
tag was created or deleted. The following additional variables are
|
||||
available:
|
||||
|
||||
.. var:: zuul
|
||||
:hidden:
|
||||
|
||||
.. var:: tag
|
||||
|
||||
The name of the item's tag (without the `refs/tags/` prefix).
|
||||
|
||||
.. var:: oldrev
|
||||
|
||||
If the item was enqueued as the result of a tag being deleted,
|
||||
the previous git sha of the tag will be included here. If the
|
||||
tag was created, this variable will be undefined.
|
||||
|
||||
.. var:: newrev
|
||||
|
||||
If the item was enqueued as the result of a tag being created,
|
||||
the new git sha of the tag will be included here. If the tag
|
||||
was deleted, this variable will be undefined.
|
||||
|
||||
Ref Items
|
||||
+++++++++
|
||||
|
||||
This represents a git reference that is neither a change, branch, or
|
||||
tag. Note that all items include a `ref` attribute which may be used
|
||||
to identify the ref. The following additional variables are
|
||||
available:
|
||||
|
||||
.. var:: zuul
|
||||
:hidden:
|
||||
|
||||
.. var:: oldrev
|
||||
|
||||
If the item was enqueued as the result of a ref being deleted,
|
||||
the previous git sha of the ref will be included here. If the
|
||||
ref was created, this variable will be undefined.
|
||||
|
||||
.. var:: newrev
|
||||
|
||||
If the item was enqueued as the result of a ref being created,
|
||||
the new git sha of the ref will be included here. If the ref
|
||||
was deleted, this variable will be undefined.
|
||||
|
||||
Working Directory
|
||||
+++++++++++++++++
|
||||
|
||||
Additionally, some information about the working directory and the
|
||||
executor running the job is available:
|
||||
|
||||
.. var:: zuul
|
||||
:hidden:
|
||||
|
||||
.. var:: executor
|
||||
|
||||
A number of values related to the executor running the job are
|
||||
available:
|
||||
|
||||
.. var:: hostname
|
||||
|
||||
The hostname of the executor.
|
||||
|
||||
.. var:: src_root
|
||||
|
||||
The path to the source directory.
|
||||
|
||||
.. var:: log_root
|
||||
|
||||
The path to the logs directory.
|
||||
|
||||
.. var:: work_root
|
||||
|
||||
The path to the working directory.
|
||||
|
||||
.. _user_sitewide_variables:
|
||||
|
||||
Site-wide Variables
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Zuul administrator may define variables which will be available to
|
||||
all jobs running in the system. These are statically defined and may
|
||||
not be altered by jobs. See the :ref:`Administrator's Guide
|
||||
<admin_sitewide_variables>` for information on how a site
|
||||
administrator may define these variables.
|
||||
|
||||
Parent Job Results
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A job may return data to Zuul for later use by jobs which depend on
|
||||
it. For details, see :ref:`return_values`.
|
||||
|
||||
SSH Keys
|
||||
--------
|
||||
|
||||
Zuul starts each job with an SSH agent running and the key used to
|
||||
access the job's nodes added to that agent. Generally you won't need
|
||||
to be aware of this since Ansible will use this when performing any
|
||||
tasks on remote nodes. However, under some circumstances you may want
|
||||
to interact with the agent. For example, you may wish to add a key
|
||||
provided as a secret to the job in order to access a specific host, or
|
||||
you may want to, in a pre-playbook, replace the key used to log into
|
||||
the assigned nodes in order to further protect it from being abused by
|
||||
untrusted job content.
|
||||
|
||||
.. TODO: describe standard lib and link to published docs for it.
|
||||
|
||||
.. _return_values:
|
||||
|
||||
Return Values
|
||||
-------------
|
||||
|
||||
A job may return some values to Zuul to affect its behavior and for
|
||||
use by other jobs.. To return a value, use the ``zuul_return``
|
||||
Ansible module in a job playbook running on the executor 'localhost' node.
|
||||
For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
tasks:
|
||||
- zuul_return:
|
||||
data:
|
||||
foo: bar
|
||||
|
||||
Will return the dictionary ``{'foo': 'bar'}`` to Zuul.
|
||||
|
||||
.. TODO: xref to section describing formatting
|
||||
|
||||
To set the log URL for a build, use *zuul_return* to set the
|
||||
**zuul.log_url** value. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
tasks:
|
||||
- zuul_return:
|
||||
data:
|
||||
zuul:
|
||||
log_url: http://logs.example.com/path/to/build/logs
|
||||
|
||||
Any values other than those in the ``zuul`` hierarchy will be supplied
|
||||
as Ansible variables to child jobs. These variables have less
|
||||
precedence than any other type of variable in Zuul, so be sure their
|
||||
names are not shared by any job variables. If more than one parent
|
||||
job returns the same variable, the value from the later job in the job
|
||||
graph will take precedence.
|
||||
1072
doc/source/zuul.rst
1072
doc/source/zuul.rst
File diff suppressed because it is too large
Load Diff
|
|
@ -6,10 +6,10 @@ pipelines:
|
|||
- event: patchset-created
|
||||
success:
|
||||
gerrit:
|
||||
verified: 1
|
||||
Verified: 1
|
||||
failure:
|
||||
gerrit:
|
||||
verified: -1
|
||||
Verified: -1
|
||||
|
||||
- name: tests
|
||||
manager: IndependentPipelineManager
|
||||
|
|
@ -19,10 +19,10 @@ pipelines:
|
|||
email_filter: ^.*@example.org$
|
||||
success:
|
||||
gerrit:
|
||||
verified: 1
|
||||
Verified: 1
|
||||
failure:
|
||||
gerrit:
|
||||
verified: -1
|
||||
Verified: -1
|
||||
|
||||
- name: post
|
||||
manager: IndependentPipelineManager
|
||||
|
|
@ -38,16 +38,16 @@ pipelines:
|
|||
gerrit:
|
||||
- event: comment-added
|
||||
approval:
|
||||
- approved: 1
|
||||
- Approved: 1
|
||||
start:
|
||||
gerrit:
|
||||
verified: 0
|
||||
Verified: 0
|
||||
success:
|
||||
gerrit:
|
||||
verified: 1
|
||||
Verified: 1
|
||||
failure:
|
||||
gerrit:
|
||||
verified: -1
|
||||
Verified: -1
|
||||
|
||||
jobs:
|
||||
- name: ^.*-merge$
|
||||
|
|
|
|||
|
|
@ -30,8 +30,10 @@ under the License.
|
|||
<script src="jquery.zuul.js"></script>
|
||||
<script src="zuul.app.js"></script>
|
||||
<script>
|
||||
// @license magnet:?xt=urn:btih:8e4f440f4c65981c5bf93c76d35135ba5064d8b7&dn=apache-2.0.txt Apache 2.0
|
||||
zuul_build_dom(jQuery, '#zuul_container');
|
||||
zuul_start(jQuery);
|
||||
// @license-end
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
// jquery plugin for Zuul status page
|
||||
//
|
||||
// @licstart The following is the entire license notice for the
|
||||
// JavaScript code in this page.
|
||||
//
|
||||
// Copyright 2012 OpenStack Foundation
|
||||
// Copyright 2013 Timo Tijhof
|
||||
// Copyright 2013 Wikimedia Foundation
|
||||
|
|
@ -16,6 +19,9 @@
|
|||
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
// License for the specific language governing permissions and limitations
|
||||
// under the License.
|
||||
//
|
||||
// @licend The above is the entire license notice
|
||||
// for the JavaScript code in this page.
|
||||
|
||||
(function ($) {
|
||||
'use strict';
|
||||
|
|
@ -47,11 +53,16 @@
|
|||
'msg_id': '#zuul_msg',
|
||||
'pipelines_id': '#zuul_pipelines',
|
||||
'queue_events_num': '#zuul_queue_events_num',
|
||||
'queue_management_events_num': '#zuul_queue_management_events_num',
|
||||
'queue_results_num': '#zuul_queue_results_num',
|
||||
}, options);
|
||||
|
||||
var collapsed_exceptions = [];
|
||||
var current_filter = read_cookie('zuul_filter_string', '');
|
||||
var change_set_in_url = window.location.href.split('#')[1];
|
||||
if (change_set_in_url) {
|
||||
current_filter = change_set_in_url;
|
||||
}
|
||||
var $jq;
|
||||
|
||||
var xhr,
|
||||
|
|
@ -86,7 +97,15 @@
|
|||
job: function(job) {
|
||||
var $job_line = $('<span />');
|
||||
|
||||
if (job.url !== null) {
|
||||
if (job.result !== null) {
|
||||
$job_line.append(
|
||||
$('<a />')
|
||||
.addClass('zuul-job-name')
|
||||
.attr('href', job.report_url)
|
||||
.text(job.name)
|
||||
);
|
||||
}
|
||||
else if (job.url !== null) {
|
||||
$job_line.append(
|
||||
$('<a />')
|
||||
.addClass('zuul-job-name')
|
||||
|
|
@ -263,29 +282,38 @@
|
|||
|
||||
change_header: function(change) {
|
||||
var change_id = change.id || 'NA';
|
||||
if (change_id.length === 40) {
|
||||
change_id = change_id.substr(0, 7);
|
||||
}
|
||||
|
||||
var $change_link = $('<small />');
|
||||
if (change.url !== null) {
|
||||
if (/^[0-9a-f]{40}$/.test(change.id)) {
|
||||
var change_id_short = change.id.slice(0, 7);
|
||||
var github_id = change_id.match(/^([0-9]+),([0-9a-f]{40})$/);
|
||||
if (github_id) {
|
||||
$change_link.append(
|
||||
$('<a />').attr('href', change.url).append(
|
||||
$('<abbr />')
|
||||
.attr('title', change.id)
|
||||
.attr('title', change_id)
|
||||
.text('#' + github_id[1])
|
||||
)
|
||||
);
|
||||
} else if (/^[0-9a-f]{40}$/.test(change_id)) {
|
||||
var change_id_short = change_id.slice(0, 7);
|
||||
$change_link.append(
|
||||
$('<a />').attr('href', change.url).append(
|
||||
$('<abbr />')
|
||||
.attr('title', change_id)
|
||||
.text(change_id_short)
|
||||
)
|
||||
);
|
||||
}
|
||||
else {
|
||||
$change_link.append(
|
||||
$('<a />').attr('href', change.url).text(change.id)
|
||||
$('<a />').attr('href', change.url).text(change_id)
|
||||
);
|
||||
}
|
||||
}
|
||||
else {
|
||||
if (change_id.length === 40) {
|
||||
change_id = change_id.substr(0, 7);
|
||||
}
|
||||
$change_link.text(change_id);
|
||||
}
|
||||
|
||||
|
|
@ -686,6 +714,10 @@
|
|||
data.trigger_event_queue ?
|
||||
data.trigger_event_queue.length : '0'
|
||||
);
|
||||
$(options.queue_management_events_num).text(
|
||||
data.management_event_queue ?
|
||||
data.management_event_queue.length : '0'
|
||||
);
|
||||
$(options.queue_results_num).text(
|
||||
data.result_event_queue ?
|
||||
data.result_event_queue.length : '0'
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
// Client script for Zuul status page
|
||||
//
|
||||
// @licstart The following is the entire license notice for the
|
||||
// JavaScript code in this page.
|
||||
//
|
||||
// Copyright 2013 OpenStack Foundation
|
||||
// Copyright 2013 Timo Tijhof
|
||||
// Copyright 2013 Wikimedia Foundation
|
||||
|
|
@ -16,18 +19,19 @@
|
|||
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
// License for the specific language governing permissions and limitations
|
||||
// under the License.
|
||||
//
|
||||
// @licend The above is the entire license notice
|
||||
// for the JavaScript code in this page.
|
||||
|
||||
/*exported zuul_build_dom, zuul_start */
|
||||
|
||||
function zuul_build_dom($, container) {
|
||||
// Build a default-looking DOM
|
||||
var default_layout = '<div class="container">'
|
||||
+ '<h1>Zuul Status</h1>'
|
||||
+ '<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>'
|
||||
+ '<div class="zuul-container" id="zuul-container">'
|
||||
+ '<div style="display: none;" class="alert" id="zuul_msg"></div>'
|
||||
+ '<button class="btn pull-right zuul-spinner">updating <span class="glyphicon glyphicon-refresh"></span></button>'
|
||||
+ '<p>Queue lengths: <span id="zuul_queue_events_num">0</span> events, <span id="zuul_queue_results_num">0</span> results.</p>'
|
||||
+ '<p>Queue lengths: <span id="zuul_queue_events_num">0</span> events, <span id="zuul_queue_management_events_num">0</span> management events, <span id="zuul_queue_results_num">0</span> results.</p>'
|
||||
+ '<div id="zuul_controls"></div>'
|
||||
+ '<div id="zuul_pipelines" class="row"></div>'
|
||||
+ '<p>Zuul version: <span id="zuul-version-span"></span></p>'
|
||||
|
|
|
|||
|
|
@ -1,34 +1,49 @@
|
|||
[gearman]
|
||||
server=127.0.0.1
|
||||
;port=4730
|
||||
;ssl_ca=/path/to/ca.pem
|
||||
;ssl_cert=/path/to/client.pem
|
||||
;ssl_key=/path/to/client.key
|
||||
|
||||
[statsd]
|
||||
server=127.0.0.1
|
||||
|
||||
[zookeeper]
|
||||
hosts=127.0.0.1:2181
|
||||
|
||||
[gearman_server]
|
||||
start=true
|
||||
;ssl_ca=/path/to/ca.pem
|
||||
;ssl_cert=/path/to/server.pem
|
||||
;ssl_key=/path/to/server.key
|
||||
;port=4730
|
||||
|
||||
[zuul]
|
||||
layout_config=/etc/zuul/layout.yaml
|
||||
[scheduler]
|
||||
tenant_config=/etc/zuul/main.yaml
|
||||
log_config=/etc/zuul/logging.conf
|
||||
pidfile=/var/run/zuul/zuul.pid
|
||||
state_dir=/var/lib/zuul
|
||||
status_url=https://jenkins.example.com/zuul/status
|
||||
|
||||
[merger]
|
||||
git_dir=/var/lib/zuul/git
|
||||
;git_user_email=zuul@example.com
|
||||
;git_user_name=zuul
|
||||
zuul_url=http://zuul.example.com/p
|
||||
|
||||
[swift]
|
||||
authurl=https://identity.api.example.org/v2.0/
|
||||
user=username
|
||||
key=password
|
||||
[executor]
|
||||
default_username=zuul
|
||||
trusted_ro_paths=/opt/zuul-scripts:/var/cache
|
||||
trusted_rw_paths=/opt/zuul-logs
|
||||
|
||||
default_container=logs
|
||||
region_name=EXP
|
||||
logserver_prefix=http://logs.example.org/server.app/
|
||||
[web]
|
||||
listen_address=127.0.0.1
|
||||
port=9000
|
||||
static_cache_expiry=0
|
||||
;sql_connection_name=mydatabase
|
||||
|
||||
[webapp]
|
||||
listen_address=0.0.0.0
|
||||
port=8001
|
||||
status_url=https://zuul.example.com/status
|
||||
|
||||
[connection gerrit]
|
||||
driver=gerrit
|
||||
|
|
|
|||
|
|
@ -0,0 +1,26 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
|
||||
- name: Collect openstack-zuul-jobs generated job config
|
||||
synchronize:
|
||||
dest: "{{ zuul.executor.log_root }}/openstack-zuul-jobs"
|
||||
mode: pull
|
||||
src: "src/git.openstack.org/openstack-infra/openstack-zuul-jobs/zuul.d"
|
||||
verify_host: true
|
||||
no_log: true
|
||||
|
||||
- name: Collect project generated job config
|
||||
synchronize:
|
||||
dest: "{{ zuul.executor.log_root }}/openstack-zuul-jobs"
|
||||
mode: pull
|
||||
src: "src/git.openstack.org/openstack-infra/project-config/zuul.d"
|
||||
verify_host: true
|
||||
no_log: true
|
||||
|
||||
- name: Collect generated playbooks
|
||||
synchronize:
|
||||
dest: "{{ zuul.executor.log_root }}/playbooks"
|
||||
mode: pull
|
||||
src: "src/git.openstack.org/openstack-infra/openstack-zuul-jobs/playbooks/legacy"
|
||||
verify_host: true
|
||||
no_log: true
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
|
||||
- name: Install migration dependencies
|
||||
command: "python3 -m pip install --user src/git.openstack.org/openstack-infra/zuul[migrate]"
|
||||
|
||||
- name: Migrate the data
|
||||
command: tools/run-migration.sh -v --final
|
||||
args:
|
||||
chdir: src/git.openstack.org/openstack-infra/zuul
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
- name: Run some commands to show that logging works on failed tasks too
|
||||
hosts: node
|
||||
tasks:
|
||||
|
||||
- block:
|
||||
|
||||
- name: Run a shell task with an ansible python exception
|
||||
command: echo foo
|
||||
args:
|
||||
chdir: /failure-shelltask/somewhere/that/does/not/exist
|
||||
|
||||
always:
|
||||
|
||||
- name: Loop with items on an ansible python exception
|
||||
command: "echo {{ item }}"
|
||||
with_items:
|
||||
- item1
|
||||
- item2
|
||||
- item3
|
||||
args:
|
||||
chdir: /failure-itemloop/somewhere/that/does/not/exist
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
- name: Run some commands to show that logging works
|
||||
hosts: node
|
||||
tasks:
|
||||
|
||||
- name: Run setup
|
||||
setup:
|
||||
register: setupvar
|
||||
|
||||
- name: Output debug for a var
|
||||
debug:
|
||||
var: setupvar
|
||||
|
||||
- name: Output a debug sentence
|
||||
debug:
|
||||
msg: This is a debug message
|
||||
|
||||
- name: Run a shell task
|
||||
command: ip addr show
|
||||
|
||||
- name: Loop with items
|
||||
command: "echo {{ item }}"
|
||||
with_items:
|
||||
- item1
|
||||
- item2
|
||||
- item3
|
||||
|
||||
- name: Loop with complex items
|
||||
command: "echo {{ item.name }}"
|
||||
with_items:
|
||||
- name: complex1
|
||||
- name: complex2
|
||||
- name: complex3
|
||||
|
||||
- name: Run a shell task with an ansible python exception
|
||||
command: echo foo
|
||||
args:
|
||||
chdir: /shelltask/somewhere/that/does/not/exist
|
||||
failed_when: false
|
||||
|
||||
- name: Loop with items on an ansible python exception
|
||||
command: "echo {{ item }}"
|
||||
with_items:
|
||||
- item1
|
||||
- item2
|
||||
- item3
|
||||
args:
|
||||
chdir: /itemloop/somewhere/that/does/not/exist
|
||||
failed_when: false
|
||||
|
||||
- name: Print binary data
|
||||
command: echo -e '\x80abc'
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
- hosts: controller
|
||||
tasks:
|
||||
|
||||
- name: Run ansible that should succeed
|
||||
command: ansible-playbook src/git.openstack.org/openstack-infra/zuul/playbooks/zuul-stream/fixtures/test-stream.yaml
|
||||
environment:
|
||||
ZUUL_JOB_LOG_CONFIG: "{{ ansible_user_dir}}/logging.json"
|
||||
ARA_LOG_CONFIG: "{{ ansible_user_dir}}/logging.json"
|
||||
|
||||
- name: Run ansible playbook that should fail
|
||||
command: ansible-playbook src/git.openstack.org/openstack-infra/zuul/playbooks/zuul-stream/fixtures/test-stream-failure.yaml
|
||||
register: failed_results
|
||||
failed_when: "failed_results.rc != 2"
|
||||
environment:
|
||||
ZUUL_JOB_LOG_CONFIG: "{{ ansible_user_dir}}/logging.json"
|
||||
ARA_LOG_CONFIG: "{{ ansible_user_dir}}/logging.json"
|
||||
|
||||
- name: Validate output - setupvar
|
||||
shell: |
|
||||
egrep "^.*\| node1 \|\s+\"setupvar\": {" job-output.txt
|
||||
egrep "^.*\| node2 \|\s+\"setupvar\": {" job-output.txt
|
||||
|
||||
- name: Validate output - shell task
|
||||
shell: |
|
||||
egrep "^.*\| node1 \| link/loopback" job-output.txt
|
||||
egrep "^.*\| node2 \| link/loopback" job-output.txt
|
||||
|
||||
- name: Validate output - loop with items
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| ok: Item: item1" job-output.txt
|
||||
egrep "^.+\| node1 \| ok: Item: item2" job-output.txt
|
||||
egrep "^.+\| node1 \| ok: Item: item3" job-output.txt
|
||||
egrep "^.+\| node2 \| ok: Item: item1" job-output.txt
|
||||
egrep "^.+\| node2 \| ok: Item: item2" job-output.txt
|
||||
egrep "^.+\| node2 \| ok: Item: item3" job-output.txt
|
||||
|
||||
- name: Validate output - loop with complex items
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| ok: Item: Runtime" job-output.txt
|
||||
egrep "^.+\| node2 \| ok: Item: Runtime" job-output.txt
|
||||
|
||||
- name: Validate output - shell task with exception
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| OSError.+\/shelltask\/" job-output.txt
|
||||
egrep "^.+\| node2 \| OSError.+\/shelltask\/" job-output.txt
|
||||
|
||||
- name: Validate output - item loop with exception
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| OSError.+\/itemloop\/" job-output.txt
|
||||
egrep "^.+\| node2 \| OSError.+\/itemloop\/" job-output.txt
|
||||
|
||||
- name: Validate output - failure shell task with exception
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| OSError.+\/failure-shelltask\/" job-output.txt
|
||||
egrep "^.+\| node2 \| OSError.+\/failure-shelltask\/" job-output.txt
|
||||
|
||||
- name: Validate output - failure item loop with exception
|
||||
shell: |
|
||||
egrep "^.+\| node1 \| OSError.+\/failure-itemloop\/" job-output.txt
|
||||
egrep "^.+\| node2 \| OSError.+\/failure-itemloop\/" job-output.txt
|
||||
|
||||
- name: Validate output - binary data
|
||||
shell: |
|
||||
egrep "^.*\| node1 \| \\\\x80abc" job-output.txt
|
||||
egrep "^.*\| node2 \| \\\\x80abc" job-output.txt
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
- hosts: controller
|
||||
tasks:
|
||||
|
||||
- name: Generate ARA html
|
||||
command: ara generate html ara-output
|
||||
|
||||
- name: Compress ARA html
|
||||
command: gzip --recursive --best ara-output
|
||||
|
||||
- name: Fetch ARA files
|
||||
synchronize:
|
||||
src: "{{ ansible_user_dir }}/ara-output"
|
||||
dest: "{{ zuul.executor.log_root }}/stream-files"
|
||||
mode: pull
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
- hosts: controller
|
||||
tasks:
|
||||
|
||||
- set_fact:
|
||||
output_dir: "{{ zuul.executor.log_root }}/stream-files"
|
||||
|
||||
- name: Make log subdir
|
||||
file:
|
||||
path: "{{ output_dir }}"
|
||||
state: directory
|
||||
delegate_to: localhost
|
||||
|
||||
- name: Rename job-output.txt
|
||||
command: mv job-output.txt stream-job-output.txt
|
||||
|
||||
- name: Fetch files
|
||||
synchronize:
|
||||
src: "{{ ansible_user_dir }}/{{ item }}"
|
||||
dest: "{{ output_dir }}"
|
||||
mode: pull
|
||||
with_items:
|
||||
- logging.json
|
||||
- ansible.cfg
|
||||
- stream-job-output.txt
|
||||
- job-output.json
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
- hosts: controller
|
||||
roles:
|
||||
|
||||
- role: bindep
|
||||
bindep_profile: test
|
||||
bindep_dir: src/git.openstack.org/openstack-infra/zuul
|
||||
|
||||
- role: bindep
|
||||
bindep_dir: src/git.openstack.org/openstack/ara
|
||||
|
||||
post_tasks:
|
||||
|
||||
- name: Install software
|
||||
command: python3 -m pip install src/git.openstack.org/openstack-infra/zuul src/git.openstack.org/openstack/ara
|
||||
become: yes
|
||||
|
||||
- name: Copy inventory
|
||||
copy:
|
||||
src: "{{ zuul.executor.log_root }}/zuul-info/inventory.yaml"
|
||||
dest: "{{ ansible_user_dir }}/inventory.yaml"
|
||||
|
||||
- name: Copy ansible.cfg
|
||||
template:
|
||||
src: templates/ansible.cfg.j2
|
||||
dest: "{{ ansible_user_dir }}/ansible.cfg"
|
||||
|
||||
- name: Generate logging config
|
||||
command: python3 src/git.openstack.org/openstack-infra/zuul/zuul/ansible/logconfig.py
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[defaults]
|
||||
inventory = {{ ansible_user_dir }}/inventory.yaml
|
||||
gathering = smart
|
||||
gather_subset = !all
|
||||
lookup_plugins = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/lookup
|
||||
action_plugins = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/action
|
||||
callback_plugins = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/callback:{{ ansible_user_dir }}/src/git.openstack.org/openstack/ara/ara/plugins/callbacks
|
||||
module_utils = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/module_utils
|
||||
stdout_callback = zuul_stream
|
||||
library = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/library
|
||||
retry_files_enabled = False
|
||||
|
|
@ -1,19 +1,29 @@
|
|||
pbr>=1.1.0
|
||||
|
||||
# pull from master until https://github.com/sigmavirus24/github3.py/pull/671
|
||||
# is in a release
|
||||
git+https://github.com/sigmavirus24/github3.py.git@develop#egg=Github3.py
|
||||
PyYAML>=3.1.0
|
||||
Paste
|
||||
WebOb>=1.2.3
|
||||
paramiko>=1.8.0,<2.0.0
|
||||
GitPython>=0.3.3,<2.1.2
|
||||
ordereddict
|
||||
GitPython>=2.1.8
|
||||
python-daemon>=2.0.4,<2.1.0
|
||||
extras
|
||||
statsd>=1.0.0,<3.0
|
||||
voluptuous>=0.10.2
|
||||
gear>=0.5.7,<1.0.0
|
||||
gear>=0.9.0,<1.0.0
|
||||
apscheduler>=3.0
|
||||
PrettyTable>=0.6,<0.8
|
||||
babel>=1.0
|
||||
six>=1.6.0
|
||||
ansible>=2.3.0.0,<2.4
|
||||
netaddr
|
||||
kazoo
|
||||
sqlalchemy
|
||||
alembic
|
||||
cryptography>=1.6
|
||||
cachecontrol
|
||||
pyjwt
|
||||
iso8601
|
||||
aiohttp
|
||||
uvloop;python_version>='3.5'
|
||||
|
|
|
|||
16
setup.cfg
16
setup.cfg
|
|
@ -12,26 +12,32 @@ classifier =
|
|||
License :: OSI Approved :: Apache Software License
|
||||
Operating System :: POSIX :: Linux
|
||||
Programming Language :: Python
|
||||
Programming Language :: Python :: 2
|
||||
Programming Language :: Python :: 2.7
|
||||
Programming Language :: Python :: 2.6
|
||||
Programming Language :: Python :: 3
|
||||
Programming Language :: Python :: 3.5
|
||||
|
||||
[pbr]
|
||||
warnerrors = True
|
||||
|
||||
[entry_points]
|
||||
console_scripts =
|
||||
zuul-server = zuul.cmd.server:main
|
||||
zuul-scheduler = zuul.cmd.scheduler:main
|
||||
zuul-merger = zuul.cmd.merger:main
|
||||
zuul = zuul.cmd.client:main
|
||||
zuul-cloner = zuul.cmd.cloner:main
|
||||
zuul-launcher = zuul.cmd.launcher:main
|
||||
zuul-executor = zuul.cmd.executor:main
|
||||
zuul-bwrap = zuul.driver.bubblewrap:main
|
||||
zuul-web = zuul.cmd.web:main
|
||||
zuul-migrate = zuul.cmd.migrate:main
|
||||
zuul-fingergw = zuul.cmd.fingergw:main
|
||||
|
||||
[build_sphinx]
|
||||
source-dir = doc/source
|
||||
build-dir = doc/build
|
||||
all_files = 1
|
||||
warning-is-error = 1
|
||||
|
||||
[extras]
|
||||
mysql_reporter=
|
||||
PyMySQL
|
||||
migrate=
|
||||
jenkins-job-builder==1.6.2
|
||||
|
|
|
|||
|
|
@ -1,14 +1,18 @@
|
|||
hacking>=0.12.0,!=0.13.0,<0.14 # Apache-2.0
|
||||
pep8
|
||||
pyflakes
|
||||
flake8
|
||||
|
||||
coverage>=3.6
|
||||
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
|
||||
sphinx>=1.5.1,<1.6
|
||||
sphinxcontrib-blockdiag>=1.1.0
|
||||
fixtures>=0.3.14
|
||||
python-keystoneclient>=0.4.2
|
||||
python-subunit
|
||||
python-swiftclient>=1.6
|
||||
testrepository>=0.0.17
|
||||
testtools>=0.9.32
|
||||
sphinxcontrib-programoutput
|
||||
sphinx-autodoc-typehints
|
||||
mock
|
||||
PyMySQL
|
||||
mypy
|
||||
zuul-sphinx
|
||||
|
|
|
|||
2616
tests/base.py
2616
tests/base.py
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,38 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import base64
|
||||
import sys
|
||||
import os
|
||||
|
||||
from zuul.lib import encryption
|
||||
|
||||
FIXTURE_DIR = os.path.join(os.path.dirname(__file__),
|
||||
'fixtures')
|
||||
|
||||
|
||||
def main():
|
||||
private_key_file = os.path.join(FIXTURE_DIR, 'private.pem')
|
||||
with open(private_key_file, "rb") as f:
|
||||
private_key, public_key = \
|
||||
encryption.deserialize_rsa_keypair(f.read())
|
||||
|
||||
plaintext = sys.argv[1].encode('utf-8')
|
||||
|
||||
ciphertext = encryption.encrypt_pkcs1_oaep(plaintext, public_key)
|
||||
print(base64.b64encode(ciphertext).decode('utf-8'))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -0,0 +1,214 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# Copyright 2018 Red Hat, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import re
|
||||
|
||||
|
||||
class FakeUser(object):
|
||||
def __init__(self, login):
|
||||
self.login = login
|
||||
self.name = "Github User"
|
||||
self.email = "github.user@example.com"
|
||||
|
||||
|
||||
class FakeBranch(object):
|
||||
def __init__(self, branch='master'):
|
||||
self.name = branch
|
||||
|
||||
|
||||
class FakeStatus(object):
|
||||
def __init__(self, state, url, description, context, user):
|
||||
self._state = state
|
||||
self._url = url
|
||||
self._description = description
|
||||
self._context = context
|
||||
self._user = user
|
||||
|
||||
def as_dict(self):
|
||||
return {
|
||||
'state': self._state,
|
||||
'url': self._url,
|
||||
'description': self._description,
|
||||
'context': self._context,
|
||||
'creator': {
|
||||
'login': self._user
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class FakeCommit(object):
|
||||
def __init__(self):
|
||||
self._statuses = []
|
||||
|
||||
def set_status(self, state, url, description, context, user):
|
||||
status = FakeStatus(
|
||||
state, url, description, context, user)
|
||||
# always insert a status to the front of the list, to represent
|
||||
# the last status provided for a commit.
|
||||
self._statuses.insert(0, status)
|
||||
|
||||
def statuses(self):
|
||||
return self._statuses
|
||||
|
||||
|
||||
class FakeRepository(object):
|
||||
def __init__(self):
|
||||
self._branches = [FakeBranch()]
|
||||
self._commits = {}
|
||||
|
||||
def branches(self, protected=False):
|
||||
if protected:
|
||||
# simulate there is no protected branch
|
||||
return []
|
||||
return self._branches
|
||||
|
||||
def create_status(self, sha, state, url, description, context,
|
||||
user='zuul'):
|
||||
# Since we're bypassing github API, which would require a user, we
|
||||
# default the user as 'zuul' here.
|
||||
commit = self._commits.get(sha, None)
|
||||
if commit is None:
|
||||
commit = FakeCommit()
|
||||
self._commits[sha] = commit
|
||||
commit.set_status(state, url, description, context, user)
|
||||
|
||||
def commit(self, sha):
|
||||
commit = self._commits.get(sha, None)
|
||||
if commit is None:
|
||||
commit = FakeCommit()
|
||||
self._commits[sha] = commit
|
||||
return commit
|
||||
|
||||
|
||||
class FakeLabel(object):
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
|
||||
class FakeIssue(object):
|
||||
def __init__(self, fake_pull_request):
|
||||
self._fake_pull_request = fake_pull_request
|
||||
|
||||
def pull_request(self):
|
||||
return FakePull(self._fake_pull_request)
|
||||
|
||||
def labels(self):
|
||||
return [FakeLabel(l)
|
||||
for l in self._fake_pull_request.labels]
|
||||
|
||||
|
||||
class FakeFile(object):
|
||||
def __init__(self, filename):
|
||||
self.filename = filename
|
||||
|
||||
|
||||
class FakePull(object):
|
||||
def __init__(self, fake_pull_request):
|
||||
self._fake_pull_request = fake_pull_request
|
||||
|
||||
def issue(self):
|
||||
return FakeIssue(self._fake_pull_request)
|
||||
|
||||
def files(self):
|
||||
return [FakeFile(fn)
|
||||
for fn in self._fake_pull_request.files]
|
||||
|
||||
def as_dict(self):
|
||||
pr = self._fake_pull_request
|
||||
connection = pr.github
|
||||
data = {
|
||||
'number': pr.number,
|
||||
'title': pr.subject,
|
||||
'url': 'https://%s/%s/pull/%s' % (
|
||||
connection.server, pr.project, pr.number
|
||||
),
|
||||
'updated_at': pr.updated_at,
|
||||
'base': {
|
||||
'repo': {
|
||||
'full_name': pr.project
|
||||
},
|
||||
'ref': pr.branch,
|
||||
},
|
||||
'mergeable': True,
|
||||
'state': pr.state,
|
||||
'head': {
|
||||
'sha': pr.head_sha,
|
||||
'repo': {
|
||||
'full_name': pr.project
|
||||
}
|
||||
},
|
||||
'merged': pr.is_merged,
|
||||
'body': pr.body
|
||||
}
|
||||
return data
|
||||
|
||||
|
||||
class FakeIssueSearchResult(object):
|
||||
def __init__(self, issue):
|
||||
self.issue = issue
|
||||
|
||||
|
||||
class FakeGithub(object):
|
||||
def __init__(self, pull_requests):
|
||||
self._pull_requests = pull_requests
|
||||
self._repos = {}
|
||||
|
||||
def user(self, login):
|
||||
return FakeUser(login)
|
||||
|
||||
def repository(self, owner, proj):
|
||||
return self._repos.get((owner, proj), None)
|
||||
|
||||
def repo_from_project(self, project):
|
||||
# This is a convenience method for the tests.
|
||||
owner, proj = project.split('/')
|
||||
return self.repository(owner, proj)
|
||||
|
||||
def addProject(self, project):
|
||||
owner, proj = project.name.split('/')
|
||||
self._repos[(owner, proj)] = FakeRepository()
|
||||
|
||||
def pull_request(self, owner, project, number):
|
||||
fake_pr = self._pull_requests[number]
|
||||
return FakePull(fake_pr)
|
||||
|
||||
def search_issues(self, query):
|
||||
def tokenize(s):
|
||||
return re.findall(r'[\w]+', s)
|
||||
|
||||
parts = tokenize(query)
|
||||
terms = set()
|
||||
results = []
|
||||
for part in parts:
|
||||
kv = part.split(':', 1)
|
||||
if len(kv) == 2:
|
||||
if kv[0] in set('type', 'is', 'in'):
|
||||
# We only perform one search now and these aren't
|
||||
# important; we can honor these terms later if
|
||||
# necessary.
|
||||
continue
|
||||
terms.add(part)
|
||||
|
||||
for pr in self._pull_requests.values():
|
||||
if not pr.body:
|
||||
body = set()
|
||||
else:
|
||||
body = set(tokenize(pr.body))
|
||||
if terms.intersection(body):
|
||||
issue = FakeIssue(pr)
|
||||
results.append(FakeIssueSearchResult(issue))
|
||||
|
||||
return results
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
- file:
|
||||
path: "{{zuul._test.test_root}}/{{zuul.build}}.bare-role.flag"
|
||||
state: touch
|
||||
10
tests/fixtures/config/ansible/git/common-config/playbooks/check-secret-names.yaml
vendored
Normal file
10
tests/fixtures/config/ansible/git/common-config/playbooks/check-secret-names.yaml
vendored
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
- hosts: ubuntu-xenial
|
||||
tasks:
|
||||
|
||||
- debug:
|
||||
msg: "renamed secret {{ renamed_secret }}"
|
||||
|
||||
- name: Assert variable precedence.
|
||||
assert:
|
||||
that:
|
||||
- renamed_secret.value == 'vartest_secret'
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
- hosts: ubuntu-xenial
|
||||
tasks:
|
||||
- name: Assert nodepool variables are valid.
|
||||
assert:
|
||||
that:
|
||||
- nodepool.az == 'test-az'
|
||||
- nodepool.cloud == 'test-cloud'
|
||||
- nodepool.region == 'test-region'
|
||||
- nodepool.provider == 'test-provider'
|
||||
|
||||
- name: Assert zuul-executor variables are valid.
|
||||
assert:
|
||||
that:
|
||||
- zuul.executor.hostname is defined
|
||||
- zuul.executor.src_root is defined
|
||||
- zuul.executor.log_root is defined
|
||||
- zuul.executor.work_root is defined
|
||||
|
||||
- name: Assert zuul.project variables are valid.
|
||||
assert:
|
||||
that:
|
||||
- zuul.project.name == 'org/project'
|
||||
- zuul.project.canonical_hostname == 'review.example.com'
|
||||
- zuul.project.canonical_name == 'review.example.com/org/project'
|
||||
- zuul.project.src_dir == 'src/review.example.com/org/project'
|
||||
|
||||
- name: Assert legacy zuul vars are valid
|
||||
assert:
|
||||
that:
|
||||
- zuul.project.name == '{{ (zuul | zuul_legacy_vars).ZUUL_PROJECT }}'
|
||||
- zuul.branch == '{{ (zuul | zuul_legacy_vars).ZUUL_BRANCH }}'
|
||||
|
||||
- debug:
|
||||
msg: "vartest secret {{ vartest_secret }}"
|
||||
|
||||
- name: Assert variable precedence.
|
||||
assert:
|
||||
that:
|
||||
- vartest_job == 'vartest_job'
|
||||
- vartest_secret.value == 'vartest_secret'
|
||||
- vartest_site == 'vartest_site'
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- name: Register hello-world.txt file.
|
||||
stat:
|
||||
path: "{{zuul.executor.log_root}}/hello-world.txt"
|
||||
register: st
|
||||
|
||||
- name: Assert hello-world.txt file.
|
||||
assert:
|
||||
that:
|
||||
- st.stat.exists
|
||||
- st.stat.isreg
|
||||
|
||||
- name: Simple shell task.
|
||||
shell: |+
|
||||
echo "Hello world"
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- shell: |+
|
||||
echo "I am broken"
|
||||
exit 1
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- file:
|
||||
path: "{{zuul._test.test_root}}/{{zuul.build}}.post.flag"
|
||||
state: touch
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- file:
|
||||
path: "{{zuul._test.test_root}}/{{zuul.build}}.pre.flag"
|
||||
state: touch
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- file:
|
||||
path: "{{flagpath}}"
|
||||
state: touch
|
||||
- copy:
|
||||
src: "{{zuul._test.test_root}}/{{zuul.build}}.flag"
|
||||
dest: "{{zuul._test.test_root}}/{{zuul.build}}.copied"
|
||||
- copy:
|
||||
content: "{{test_secret.username}} {{test_secret.password}}"
|
||||
dest: "{{zuul._test.test_root}}/{{zuul.build}}.secrets"
|
||||
roles:
|
||||
- bare-role
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- name: Pause for 60 seconds, so zuul aborts our job.
|
||||
shell: sleep 60
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
- pipeline:
|
||||
name: check
|
||||
manager: independent
|
||||
post-review: true
|
||||
trigger:
|
||||
gerrit:
|
||||
- event: patchset-created
|
||||
success:
|
||||
gerrit:
|
||||
Verified: 1
|
||||
failure:
|
||||
gerrit:
|
||||
Verified: -1
|
||||
|
||||
- pipeline:
|
||||
name: gate
|
||||
manager: dependent
|
||||
success-message: Build succeeded (gate).
|
||||
trigger:
|
||||
gerrit:
|
||||
- event: comment-added
|
||||
approval:
|
||||
- Approved: 1
|
||||
success:
|
||||
gerrit:
|
||||
Verified: 2
|
||||
submit: true
|
||||
failure:
|
||||
gerrit:
|
||||
Verified: -2
|
||||
start:
|
||||
gerrit:
|
||||
Verified: 0
|
||||
precedence: high
|
||||
|
||||
- secret:
|
||||
name: test_secret
|
||||
data:
|
||||
username: test-username
|
||||
password: !encrypted/pkcs1-oaep |
|
||||
BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi1wGZZ
|
||||
L0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4joeusC9drN3AA8a4o
|
||||
ykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CRgd0QBMPl6VDoFgBPB8vxtJw+
|
||||
3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzibDsSXsfJt1y+5n7yOURsC7lovMg4GF/v
|
||||
Cl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCYceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qt
|
||||
xhbpjTxG4U5Q/SoppOJ60WqEkQvbXs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYr
|
||||
aI+AKYsMYx3RBlfAmCeC1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFW
|
||||
Z3QSO1NjbBxWnaHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd
|
||||
+150AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZHvIs=
|
||||
|
||||
# This is used by the check-vars job to evaluate variable precedence.
|
||||
# The name of this secret conflicts with a site variable.
|
||||
- secret:
|
||||
name: vartest_site
|
||||
data:
|
||||
value: vartest_secret
|
||||
|
||||
# This is used by the check-vars job to evaluate variable precedence.
|
||||
# The name of this secret conflicts with a job variable.
|
||||
- secret:
|
||||
name: vartest_job
|
||||
data:
|
||||
value: vartest_secret
|
||||
|
||||
# This is used by the check-vars job to evaluate variable precedence.
|
||||
# The name of this secret should not conflict.
|
||||
- secret:
|
||||
name: vartest_secret
|
||||
data:
|
||||
value: vartest_secret
|
||||
|
||||
- job:
|
||||
name: base
|
||||
parent: null
|
||||
|
||||
- job:
|
||||
name: base-urls
|
||||
success-url: https://success.example.com/zuul-logs/{build.uuid}/
|
||||
failure-url: https://failure.example.com/zuul-logs/{build.uuid}/
|
||||
|
||||
- job:
|
||||
name: python27
|
||||
parent: base-urls
|
||||
run: playbooks/python27.yaml
|
||||
pre-run: playbooks/pre.yaml
|
||||
post-run: playbooks/post.yaml
|
||||
vars:
|
||||
flagpath: '{{zuul._test.test_root}}/{{zuul.build}}.flag'
|
||||
roles:
|
||||
- zuul: bare-role
|
||||
secrets:
|
||||
- test_secret
|
||||
|
||||
- job:
|
||||
parent: python27
|
||||
name: timeout
|
||||
run: playbooks/timeout.yaml
|
||||
timeout: 1
|
||||
|
||||
- job:
|
||||
parent: python27
|
||||
name: check-vars
|
||||
run: playbooks/check-vars.yaml
|
||||
nodeset:
|
||||
nodes:
|
||||
- name: ubuntu-xenial
|
||||
label: ubuntu-xenial
|
||||
vars:
|
||||
vartest_job: vartest_job
|
||||
vartest_secret: vartest_job
|
||||
vartest_site: vartest_job
|
||||
secrets:
|
||||
- vartest_site
|
||||
- vartest_secret
|
||||
|
||||
- job:
|
||||
parent: python27
|
||||
name: check-secret-names
|
||||
run: playbooks/check-secret-names.yaml
|
||||
nodeset:
|
||||
nodes:
|
||||
- name: ubuntu-xenial
|
||||
label: ubuntu-xenial
|
||||
secrets:
|
||||
- secret: vartest_secret
|
||||
name: renamed_secret
|
||||
|
||||
- job:
|
||||
parent: base-urls
|
||||
name: hello
|
||||
run: playbooks/hello-post.yaml
|
||||
post-run: playbooks/hello-post.yaml
|
||||
|
||||
- job:
|
||||
parent: python27
|
||||
name: failpost
|
||||
run: playbooks/post-broken.yaml
|
||||
post-run: playbooks/post-broken.yaml
|
||||
|
|
@ -0,0 +1 @@
|
|||
test
|
||||
3
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/block_local_override.yaml
vendored
Normal file
3
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/block_local_override.yaml
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
- hosts: localhost
|
||||
roles:
|
||||
- test-local-override
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/cartesian.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/cartesian.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
vars:
|
||||
value: "{{ lookup('cartesian', [1, 2], [3, 4]) }}"
|
||||
tasks:
|
||||
- debug: msg="value is {{ value }}"
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/consul_kv.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/consul_kv.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- debug: msg='key contains {{item}}'
|
||||
with_consul_kv:
|
||||
- 'key/to/retrieve'
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/credstash.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/credstash.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- debug: msg='key contains {{item}}'
|
||||
with_credstash:
|
||||
- 'key'
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/csvfile_bad.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/csvfile_bad.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
vars:
|
||||
value: "{{ lookup('csvfile', 'a file=/etc/passwd') }}"
|
||||
tasks:
|
||||
- debug: msg="value is {{ value }}"
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/csvfile_good.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/csvfile_good.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
vars:
|
||||
value: "{{ lookup('csvfile', 'a file=test.csv delimiter=,') }}"
|
||||
tasks:
|
||||
- debug: msg="value is {{ value }}"
|
||||
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/file_local_bad.yaml
vendored
Normal file
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/file_local_bad.yaml
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
- hosts: localhost
|
||||
tasks:
|
||||
- name: Try to verify a file in a bad location
|
||||
file:
|
||||
dest: /tmp/unreadable
|
||||
state: absent
|
||||
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/file_local_good.yaml
vendored
Normal file
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/file_local_good.yaml
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
- hosts: localhost
|
||||
tasks:
|
||||
- name: Try to verify a file in an ok location
|
||||
file:
|
||||
dest: non-existent
|
||||
state: absent
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
vars:
|
||||
value: "{{ lookup('file', '/etc/passwd') }}"
|
||||
tasks:
|
||||
- debug: msg="value is {{ value }}"
|
||||
|
|
@ -1,7 +1,4 @@
|
|||
#!/usr/bin/python
|
||||
|
||||
# Copyright (c) 2016 IBM Corp.
|
||||
# Copyright (c) 2016 Red Hat
|
||||
# Copyright (c) 2017 Red Hat
|
||||
#
|
||||
# This module is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
|
|
@ -16,43 +13,19 @@
|
|||
# You should have received a copy of the GNU General Public License
|
||||
# along with this software. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
import datetime
|
||||
|
||||
|
||||
class Console(object):
|
||||
def __enter__(self):
|
||||
self.logfile = open('/tmp/console.html', 'a', 0)
|
||||
return self
|
||||
|
||||
def __exit__(self, etype, value, tb):
|
||||
self.logfile.close()
|
||||
|
||||
def addLine(self, ln):
|
||||
ts = datetime.datetime.now()
|
||||
outln = '%s | %s' % (str(ts), ln)
|
||||
self.logfile.write(outln)
|
||||
|
||||
|
||||
def log(msg):
|
||||
if not isinstance(msg, list):
|
||||
msg = [msg]
|
||||
with Console() as console:
|
||||
for line in msg:
|
||||
console.addLine("[Zuul] %s\n" % line)
|
||||
# This file, by existing, should be found instead of ansible's built in
|
||||
# file module.
|
||||
|
||||
|
||||
def main():
|
||||
module = AnsibleModule(
|
||||
argument_spec=dict(
|
||||
msg=dict(required=True, type='raw'),
|
||||
path=dict(required=False, type='str'),
|
||||
state=dict(required=False, type='dict'),
|
||||
)
|
||||
)
|
||||
|
||||
p = module.params
|
||||
log(p['msg'])
|
||||
module.exit_json(changed=True)
|
||||
module.exit_json(changed=False)
|
||||
|
||||
from ansible.module_utils.basic import * # noqa
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
from ansible.module_utils.basic import AnsibleModule
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
- name: Attempt to use local version of file.py
|
||||
file:
|
||||
path: some-file.out
|
||||
state: touch
|
||||
|
|
@ -0,0 +1 @@
|
|||
a,b,c
|
||||
|
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/uri_bad_path.yaml
vendored
Normal file
6
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/uri_bad_path.yaml
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
- hosts: localhost
|
||||
tasks:
|
||||
- uri:
|
||||
method: GET
|
||||
url: https://example.com
|
||||
path: /tmp/example.out
|
||||
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/uri_bad_scheme.yaml
vendored
Normal file
5
tests/fixtures/config/ansible/git/org_plugin-project/playbooks/uri_bad_scheme.yaml
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: localhost
|
||||
tasks:
|
||||
- uri:
|
||||
method: GET
|
||||
url: file:///etc/passwd
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
- job:
|
||||
parent: python27
|
||||
name: faillocal
|
||||
run: playbooks/faillocal.yaml
|
||||
|
||||
- job:
|
||||
parent: hello
|
||||
name: hello-world
|
||||
run: playbooks/hello-world.yaml
|
||||
|
||||
- project:
|
||||
name: org/project
|
||||
check:
|
||||
jobs:
|
||||
- python27
|
||||
- faillocal
|
||||
- check-vars
|
||||
- check-secret-names
|
||||
- timeout
|
||||
- hello-world
|
||||
- failpost
|
||||
|
|
@ -0,0 +1 @@
|
|||
test
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- copy:
|
||||
src: "{{zuul._test.test_root}}/{{zuul.build}}.flag"
|
||||
dest: "{{zuul._test.test_root}}/{{zuul.build}}.failed"
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
- hosts: all
|
||||
tasks:
|
||||
- copy:
|
||||
content: "hello world"
|
||||
dest: "{{zuul.executor.log_root}}/hello-world.txt"
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
- tenant:
|
||||
name: tenant-one
|
||||
source:
|
||||
gerrit:
|
||||
config-projects:
|
||||
- common-config
|
||||
untrusted-projects:
|
||||
- org/project
|
||||
- org/plugin-project
|
||||
- bare-role
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue