60 KiB
- title
-
Project Configuration
Project Configuration
The following sections describe the main part of Zuul's configuration. All of what follows is found within files inside of the repositories that Zuul manages.
Security Contexts
When a system administrator configures Zuul to operate on a project, they specify one of two security contexts for that project. A config-project is one which is primarily tasked with holding configuration information and job content for Zuul. Jobs which are defined in a config-project are run with elevated privileges, and all Zuul configuration items are available for use. Base jobs (that is, jobs without a parent) may only be defined in config-projects. It is expected that changes to config-projects will undergo careful scrutiny before being merged.
An untrusted-project is a project whose primary focus is not to operate Zuul, but rather it is one of the projects being tested or deployed. The Zuul configuration language available to these projects is somewhat restricted (as detailed in individual sections below), and jobs defined in these projects run in a restricted execution environment since they may be operating on changes which have not yet undergone review.
Configuration Loading
When Zuul starts, it examines all of the git repositories which are
specified by the system administrator in tenant-config
and searches for files in the root of
each repository. Zuul looks first for a file named
zuul.yaml
or a directory named zuul.d
, and if
they are not found, .zuul.yaml
or .zuul.d
(with a leading dot). In the case of an untrusted-project
, the configuration from every
branch is included, however, in the case of a config-project
, only the
master
branch is examined.
When a change is proposed to one of these files in an untrusted-project, the configuration proposed in the change is merged into the running configuration so that any changes to Zuul's configuration are self-testing as part of that change. If there is a configuration error, no jobs will be run and the error will be reported by any applicable pipelines. In the case of a change to a config-project, the new configuration is parsed and examined for errors, but the new configuration is not used in testing the change. This is because configuration in config-projects is able to access elevated privileges and should always be reviewed before being merged.
As soon as a change containing a Zuul configuration change merges to any Zuul-managed repository, the new configuration takes effect immediately.
Configuration Items
The zuul.yaml
and .zuul.yaml
configuration
files are YAML-formatted and are structured as a series of items, each
of which is described below.
In the case of a zuul.d
directory, Zuul recurses the
directory and extends the configuration using all the .yaml files in the
sorted path order. For example, to keep job's variants in a separate
file, it needs to be loaded after the main entries, for example using
number prefixes in file's names:
* zuul.d/pipelines.yaml
* zuul.d/projects.yaml
* zuul.d/01_jobs.yaml
* zuul.d/02_jobs-variants.yaml
Pipeline
A pipeline describes a workflow operation in Zuul. It associates jobs for a given project with triggering and reporting events.
Its flexible configuration allows for characterizing any number of workflows, and by specifying each as a named configuration, makes it easy to apply similar workflow operations to projects or groups of projects.
By way of example, one of the primary uses of Zuul is to perform
project gating. To do so, one can create a gate
pipeline which tells
Zuul that when a certain event (such as approval by a code reviewer)
occurs, the corresponding change or pull request should be enqueued into
the pipeline. When that happens, the jobs which have been configured to
run for that project in the gate pipeline are run, and when they
complete, the pipeline reports the results to the user.
Pipeline configuration items may only appear in config-projects
<config-project>
.
Generally, a Zuul administrator would define a small number of pipelines which represent the workflow processes used in their environment. Each project can then be added to the available pipelines as appropriate.
Here is an example check
pipeline, which runs whenever a new patchset is
created in Gerrit. If the associated jobs all report success, the
pipeline reports back to Gerrit with Verified
vote of +1,
or if at least one of them fails, a -1:
- pipeline:
name: check
manager: independent
trigger:
my_gerrit:
- event: patchset-created
success:
my_gerrit:
Verified: 1
failure:
my_gerrit:
Verified: -1
pipeline
The attributes available on a pipeline are as follows (all are optional unless otherwise specified):
name
This is used later in the project definition to indicate what jobs should be run for events in the pipeline.
manager
There are three schemes for managing pipelines:
independent
Every event in this pipeline should be treated as independent of other events in the pipeline. This is appropriate when the order of events in the pipeline doesn't matter because the results of the actions this pipeline performs can not affect other events in the pipeline. For example, when a change is first uploaded for review, you may want to run tests on that change to provide early feedback to reviewers. At the end of the tests, the change is not going to be merged, so it is safe to run these tests in parallel without regard to any other changes in the pipeline. They are independent.
Another type of pipeline that is independent is a post-merge pipeline. In that case, the changes have already merged, so the results can not affect any other events in the pipeline.
dependent
The dependent pipeline manager is designed for gating. It ensures that every change is tested exactly as it is going to be merged into the repository. An ideal gating system would test one change at a time, applied to the tip of the repository, and only if that change passed tests would it be merged. Then the next change in line would be tested the same way. In order to achieve parallel testing of changes, the dependent pipeline manager performs speculative execution on changes. It orders changes based on their entry into the pipeline. It begins testing all changes in parallel, assuming that each change ahead in the pipeline will pass its tests. If they all succeed, all the changes can be tested and merged in parallel. If a change near the front of the pipeline fails its tests, each change behind it ignores whatever tests have been completed and are tested again without the change in front. This way gate tests may run in parallel but still be tested correctly, exactly as they will appear in the repository when merged.
For more detail on the theory and operation of Zuul's dependent
pipeline manager, see: gating
.
supercedent
This is like an independent pipeline, in that every item is distinct, except that items are grouped by project and ref, and only one item for each project-ref is processed at a time. If more than one additional item is enqueued for the project-ref, previously enqueued items which have not started processing are removed.
In other words, this pipeline manager will only run jobs for the most recent item enqueued for a given project-ref.
This may be useful for post-merge pipelines which perform artifact builds where only the latest version is of use. In these cases, build resources can be conserved by avoiding building intermediate versions.
post-review
This is a boolean which indicates that this pipeline executes code
that has been reviewed. Some jobs perform actions which should not be
permitted with unreviewed code. When this value is false
those jobs will not be permitted to run in the pipeline. If a pipeline
is designed only to be used after changes are reviewed or merged, set
this value to true
to permit such jobs.
For more information, see secret
and job.post-review
.
description
This field may be used to provide a textual description of the pipeline. It may appear in the status page or in documentation.
variant-description
This field may be used to provide a textual description of the variant. It may appear in the status page or in documentation.
success-message
The introductory text in reports when all the voting jobs are successful.
failure-message
The introductory text in reports when at least one voting job fails.
merge-failure-message
The introductory text in the message reported when a change fails to merge with the current state of the repository. Defaults to "Merge failed."
footer-message
Supplies additional information after test results. Useful for adding information about the CI system such as debugging and contact details.
trigger
At least one trigger source must be supplied for each pipeline. Triggers are not exclusive -- matching events may be placed in multiple pipelines, and they will behave independently in each of the pipelines they match.
Triggers are loaded from their connection name. The driver type of
the connection will dictate which options are available. See drivers
.
require
If this section is present, it establishes prerequisites for any kind of item entering the Pipeline. Regardless of how the item is to be enqueued (via any trigger or automatic dependency resolution), the conditions specified here must be met or the item will not be enqueued. These requirements may vary depending on the source of the item being enqueued.
Requirements are loaded from their connection name. The driver type
of the connection will dictate which options are available. See drivers
.
reject
If this section is present, it establishes prerequisites that can
block an item from being enqueued. It can be considered a negative
version of pipeline.require
.
Requirements are loaded from their connection name. The driver type
of the connection will dictate which options are available. See drivers
.
dequeue-on-new-patchset
Normally, if a new patchset is uploaded to a change that is in a
pipeline, the existing entry in the pipeline will be removed (with jobs
canceled and any dependent changes that can no longer merge as well. To
suppress this behavior (and allow jobs to continue running), set this to
false
.
ignore-dependencies
In any kind of pipeline (dependent or independent), Zuul will attempt
to enqueue all dependencies ahead of the current change so that they are
tested together (independent pipelines report the results of each change
regardless of the results of changes ahead). To ignore dependencies
completely in an independent pipeline, set this to true
.
This option is ignored by dependent pipelines.
precedence
Indicates how the build scheduler should prioritize jobs for
different pipelines. Each pipeline may have one precedence, jobs for
pipelines with a higher precedence will be run before ones with lower.
The value should be one of high
, normal
, or
low
. Default: normal
.
The following options configure reporters <reporter>
. Reporters are
complementary to triggers; where a trigger is an event on a connection
which causes Zuul to enqueue an item, a reporter is the action performed
on a connection when an item is dequeued after its jobs complete. The
actual syntax for a reporter is defined by the driver which implements
it. See drivers
for
more information.
success
Describes where Zuul should report to if all the jobs complete
successfully. This section is optional; if it is omitted, Zuul will run
jobs and do nothing on success -- it will not report at all. If the
section is present, the listed reporters
<reporter>
will be asked to report on the jobs. The
reporters are listed by their connection name. The options available
depend on the driver for the supplied connection.
failure
These reporters describe what Zuul should do if at least one job fails.
merge-failure
These reporters describe what Zuul should do if it is unable to merge
in the patchset. If no merge-failure reporters are listed then the
failure
reporters will be used to notify of unsuccessful
merges.
start
These reporters describe what Zuul should do when a change is added to the pipeline. This can be used, for example, to reset a previously reported result.
disabled
These reporters describe what Zuul should do when a pipeline is
disabled. See disable-after-consecutive-failures
.
The following options can be used to alter Zuul's behavior to mitigate situations in which jobs are failing frequently (perhaps due to a problem with an external dependency, or unusually high non-deterministic test failures).
disable-after-consecutive-failures
If set, a pipeline can enter a disabled state if too many changes in a row fail. When this value is exceeded the pipeline will stop reporting to any of the success, failure or merge-failure reporters and instead only report to the disabled reporters. (No start reports are made when a pipeline is disabled).
window
Dependent pipeline managers only. Zuul can rate limit dependent
pipelines in a manner similar to TCP flow control. Jobs are only started
for items in the queue if they are within the actionable window for the
pipeline. The initial length of this window is configurable with this
value. The value given should be a positive integer value. A value of
0
disables rate limiting on the dependent pipeline manager
<pipeline.manager.dependent>
.
window-floor
Dependent pipeline managers only. This is the minimum value for the window described above. Should be a positive non zero integer value.
window-increase-type
Dependent pipeline managers only. This value describes how the window should grow when changes are successfully merged by zuul.
linear
Indicates that window-increase-factor should be added to the previous window value.
exponential
Indicates that window-increase-factor should be multiplied against the previous window value and the result will become the window size.
window-increase-factor
Dependent pipeline managers only. The value to be added or multiplied against the previous window value to determine the new window after successful change merges.
window-decrease-type
Dependent pipeline managers only. This value describes how the window should shrink when changes are not able to be merged by Zuul.
linear
Indicates that window-decrease-factor should be subtracted from the previous window value.
exponential
Indicates that window-decrease-factor should be divided against the previous window value and the result will become the window size.
window-decrease-factor
Dependent pipeline managers
<pipeline.manager.dependent>
only. The value to be
subtracted or divided against the previous window value to determine the
new window after unsuccessful change merges.
Job
A job is a unit of work performed by Zuul on an item enqueued into a pipeline. Items may run any number of jobs (which may depend on each other). Each job is an invocation of an Ansible playbook with a specific inventory of hosts. The actual tasks that are run by the job appear in the playbook for that job while the attributes that appear in the Zuul configuration specify information about when, where, and how the job should be run.
Jobs in Zuul support inheritance. Any job may specify a single parent
job, and any attributes not set on the child job are collected from the
parent job. In this way, a configuration structure may be built starting
with very basic jobs which describe characteristics that all jobs on the
system should have, progressing through stages of specialization before
arriving at a particular job. A job may inherit from any other job in
any project (however, if the other job is marked as job.final
, jobs may not
inherit from it).
A job with no parent is called a base job and may only be
defined in a config-project
. Every other job must have a parent,
and so ultimately, all jobs must have an inheritance path which
terminates at a base job. Each tenant has a default parent job which
will be used if no explicit parent is specified.
Multiple job definitions with the same name are called variants.
These may have different selection criteria which indicate to Zuul that,
for instance, the job should behave differently on a different git
branch. Unlike inheritance, all job variants must be defined in the same
project. Some attributes of jobs marked job.final
may not be overidden
When Zuul decides to run a job, it performs a process known as freezing the job. Because any number of job variants may be applicable, Zuul collects all of the matching variants and applies them in the order they appeared in the configuration. The resulting frozen job is built from attributes gathered from all of the matching variants. In this way, exactly what is run is dependent on the pipeline, project, branch, and content of the item.
In addition to the job's main playbook, each job may specify one or more pre- and post-playbooks. These are run, in order, before and after (respectively) the main playbook. They may be used to set up and tear down resources needed by the main playbook. When combined with inheritance, they provide powerful tools for job construction. A job only has a single main playbook, and when inheriting from a parent, the child's main playbook overrides (or replaces) the parent's. However, the pre- and post-playbooks are appended and prepended in a nesting fashion. So if a parent job and child job both specified pre and post playbooks, the sequence of playbooks run would be:
- parent pre-run playbook
- child pre-run playbook
- child playbook
- child post-run playbook
- parent post-run playbook
Further inheritance would nest even deeper.
Here is an example of two job definitions:
- job:
name: base
pre-run: copy-git-repos
post-run: copy-logs
- job:
name: run-tests
parent: base
nodeset:
nodes:
- name: test-node
label: fedora
job
The following attributes are available on a job; all are optional unless otherwise specified:
name
The name of the job. By default, Zuul looks for a playbook with this name to use as the main playbook for the job. This name is also referenced later in a project pipeline configuration.
parent
Specifies a job to inherit from. The parent job can be defined in
this or any other project. Any attributes not specified on a job will be
collected from its parent. If no value is supplied here, the job
specified by tenant.default-parent
will be used. If
parent is set to null
(which is only valid
in a config-project
),
this is a base job
.
description
A textual description of the job. Not currently used directly by Zuul, but it is used by the zuul-sphinx extension to Sphinx to auto-document Zuul jobs (in which case it is interpreted as ReStructuredText.
final
To prevent other jobs from inheriting from this job, and also to
prevent changing execution-related attributes when this job is specified
in a project's pipeline, set this attribute to true
.
protected
When set to true
only jobs defined in the same project
may inherit from this job. Once this is set to true
it
cannot be reset to false
.
abstract
To indicate a job is not intended to be run directly, but instead
must be inherited from, set this attribute to true
.
success-message
Normally when a job succeeds, the string SUCCESS
is
reported as the result for the job. If set, this option may be used to
supply a different string.
failure-message
Normally when a job fails, the string FAILURE
is
reported as the result for the job. If set, this option may be used to
supply a different string.
success-url
When a job succeeds, this URL is reported along with the result. If
this value is not supplied, Zuul uses the content of the job return value <return_values>
zuul.log_url. This is recommended as it allows the code
which stores the URL to the job artifacts to report exactly where they
were stored. To override this value, or if it is not set, supply an
absolute URL in this field. If a relative URL is supplied in this field,
and zuul.log_url is set, then the two will be combined
to produce the URL used for the report. This can be used to specify that
certain jobs should "deep link" into the stored job artifacts.
failure-url
When a job fails, this URL is reported along with the result. Otherwise behaves the same as success-url.
hold-following-changes
In a dependent pipeline, this option may be used to indicate that no jobs should start on any items which depend on the current item until this job has completed successfully. This may be used to conserve build resources, at the expense of inhibiting the parallelization which speeds the processing of items in a dependent pipeline.
voting
Indicates whether the result of this job should be used in determining the overall result of the item.
semaphore
The name of a semaphore
which should be acquired and released when
the job begins and ends. If the semaphore is at maximum capacity, then
Zuul will wait until it can be acquired before starting the job.
tags
Metadata about this job. Tags are units of information attached to the job; they do not affect Zuul's behavior, but they can be used within the job to characterize the job. For example, a job which tests a certain subsystem could be tagged with the name of that subsystem, and if the job's results are reported into a database, then the results of all jobs affecting that subsystem could be queried. This attribute is specified as a list of strings, and when inheriting jobs or applying variants, tags accumulate in a set, so the result is always a set of all the tags from all the jobs and variants used in constructing the frozen job, with no duplication.
secrets
A list of secrets which may be used by the job. A secret
is a named collection
of private information defined separately in the configuration. The
secrets that appear here must be defined in the same project as this job
definition.
Each item in the list may may be supplied either as a string, in
which case it references the name of a secret
definition, or as a dict. If an element in this
list is given as a dict, it must have the following fields.
name
The name to use for the Ansible variable into which the secret content will be placed.
secret
The name to use to find the secret's definition in the configuration.
For example:
- secret:
name: important-secret
data:
key: encrypted-secret-key-data
- job:
name: amazing-job:
secrets:
- name: ssh_key
secret: important-secret
will result in the following being passed as a variable to the
playbooks in amazing-job
:
ssh_key:
key: descrypted-secret-key-data
nodeset
The nodes which should be supplied to the job. This parameter may be
supplied either as a string, in which case it references a nodeset
definition which
appears elsewhere in the configuration, or a dictionary, in which case
it is interpreted in the same way as a Nodeset definition, though the
name
attribute should be omitted (in essence, it is an
anonymous Nodeset definition unique to this job). See the nodeset
reference for the
syntax to use in that case.
If a job has an empty or no nodeset definition, it will still run and may be able to perform actions on the Zuul executor.
override-checkout
When Zuul runs jobs for a proposed change, it normally checks out the
branch associated with that change on every project present in the job.
If jobs are running on a ref (such as a branch tip or tag), then that
ref is normally checked out. This attribute is used to override that
behavior and indicate that this job should, regardless of the branch for
the queue item, use the indicated ref (i.e., branch or tag) instead.
This can be used, for example, to run a previous version of the software
(from a stable maintenance branch) under test even if the change being
tested applies to a different branch (this is only likely to be useful
if there is some cross-branch interaction with some component of the
system being tested). See also the project-specific job.required-projects.override-checkout
attribute to
apply this behavior to a subset of a job's projects.
This value is also used to help select which variants of a job to
run. If override-checkout
is set, then Zuul will use this
value instead of the branch of the item being tested when collecting
jobs to run.
timeout
The time in seconds that the job should be allowed to run before it is automatically aborted and failure is reported. If no timeout is supplied, the job may run indefinitely. Supplying a timeout is highly recommended.
This timeout only applies to the pre-run and run playbooks in a job.
post-timeout
The time in seconds that each post playbook should be allowed to run before it is automatically aborted and failure is reported. If no post-timeout is supplied, the job may run indefinitely. Supplying a post-timeout is highly recommended.
The post-timeout is handled separately from the above timeout because the post playbooks are typically where you will copy jobs logs. In the event of the pre-run or run playbooks timing out we want to do our best to copy the job logs in the post-run playbooks.
attempts
When Zuul encounters an error running a job's pre-run playbook, Zuul will stop and restart the job. Errors during the main or post-run -playbook phase of a job are not affected by this parameter (they are reported immediately). This parameter controls the number of attempts to make before an error is reported.
pre-run
The name of a playbook or list of playbooks to run before the main body of a job. The full path to the playbook in the repo where the job is defined is expected.
When a job inherits from a parent, the child's pre-run playbooks are
run after the parent's. See job
for more information.
post-run
The name of a playbook or list of playbooks to run after the main body of a job. The full path to the playbook in the repo where the job is defined is expected.
When a job inherits from a parent, the child's post-run playbooks are
run before the parent's. See job
for more information.
run
The name of the main playbook for this job. If it is not supplied, the parent's playbook will be used (and likewise up the inheritance chain). The full path within the repo is required. Example:
run: playbooks/job-playbook.yaml
roles
A list of Ansible roles to prepare for the job. Because a job runs an Ansible playbook, any roles which are used by the job must be prepared and installed by Zuul before the job begins. This value is a list of dictionaries, each of which indicates one of two types of roles: a Galaxy role, which is simply a role that is installed from Ansible Galaxy, or a Zuul role, which is a role provided by a project managed by Zuul. Zuul roles are able to benefit from speculative merging and cross-project dependencies when used by playbooks in untrusted projects. Roles are added to the Ansible role path in the order they appear on the job -- roles earlier in the list will take precedence over those which follow.
In the case of job inheritance or variance, the roles used for each of the playbooks run by the job will be only those which were defined along with that playbook. If a child job inherits from a parent which defines a pre and post playbook, then the pre and post playbooks it inherits from the parent job will run only with the roles that were defined on the parent. If the child adds its own pre and post playbooks, then any roles added by the child will be available to the child's playbooks. This is so that a job which inherits from a parent does not inadvertently alter the behavior of the parent's playbooks by the addition of conflicting roles. Roles added by a child will appear before those it inherits from its parent.
If a project used for a Zuul role has branches, the usual process of
selecting which branch should be checked out applies. See job.override-checkout
for a
description of that process and how to override it. As a special case,
if the role project is the project in which this job definition appears,
then the branch in which this definition appears will be used. In other
words, a playbook may not use a role from a different branch of the same
project.
A project which supplies a role may be structured in one of two
configurations: a bare role (in which the role exists at the root of the
project), or a contained role (in which the role exists within the
roles/
directory of the project, perhaps along with other
roles). In the case of a contained role, the roles/
directory of the project is added to the role search path. In the case
of a bare role, the project itself is added to the role search path. In
case the name of the project is not the name under which the role should
be installed (and therefore referenced from Ansible), the
name
attribute may be used to specify an alternate.
A job automatically has the project in which it is defined added to
the roles path if that project appears to contain a role or
roles/
directory. By default, the project is added to the
path under its own name, however, that may be changed by explicitly
listing the project in the roles list in the usual way.
Note
Galaxy roles are not yet implemented.
galaxy
The name of the role in Ansible Galaxy. If this attribute is
supplied, Zuul will search Ansible Galaxy for a role by this name and
install it. Mutually exclusive with zuul
; either
galaxy
or zuul
must be supplied.
zuul
The name of a Zuul project which supplies the role. Mutually
exclusive with galaxy
; either galaxy
or
zuul
must be supplied.
name
The installation name of the role. In the case of a bare role, the role will be made available under this name. Ignored in the case of a contained role.
required-projects
A list of other projects which are used by this job. Any Zuul projects specified here will also be checked out by Zuul into the working directory for the job. Speculative merging and cross-repo dependencies will be honored.
The format for this attribute is either a list of strings or dictionaries. Strings are interpreted as project names, dictionaries, if used, may have the following attributes:
name
The name of the required project.
override-checkout
When Zuul runs jobs for a proposed change, it normally checks out the
branch associated with that change on every project present in the job.
If jobs are running on a ref (such as a branch tip or tag), then that
ref is normally checked out. This attribute is used to override that
behavior and indicate that this job should, regardless of the branch for
the queue item, use the indicated ref (i.e., branch or tag) instead, for
only this project. See also the job.override-checkout
attribute to apply the same
behavior to all projects in a job.
This value is also used to help select which variants of a job to
run. If override-checkout
is set, then Zuul will use this
value instead of the branch of the item being tested when collecting any
jobs to run which are defined in this project.
vars
A dictionary of variables to supply to Ansible. When inheriting from a job (or creating a variant of a job) vars are merged with previous definitions. This means a variable definition with the same name will override a previously defined variable, but new variable names will be added to the set of defined variables.
extra-vars
A dictionary of variables to be passed to ansible command-line using the --extra-vars flag. Note by using extra-vars, these variables always win precedence.
host-vars
A dictionary of host variables to supply to Ansible. The keys of this
dictionary are node names as defined in a nodeset
, and the values are dictionaries of variables,
just as in job.vars
.
group-vars
A dictionary of group variables to supply to Ansible. The keys of
this dictionary are node groups as defined in a nodeset
, and the values are
dictionaries of variables, just as in job.vars
.
An example of three kinds of variables:
- job:
name: variable-example
nodeset:
nodes:
- name: controller
label: fedora-27
- name: api1
label: centos-7
- name: api2
label: centos-7
groups:
- name: api
nodes:
- api1
- api2
vars:
foo: "this variable is visible to all nodes"
host-vars:
controller:
bar: "this variable is visible only on the controller node"
group-vars:
api:
baz: "this variable is visible on api1 and api2"
dependencies
A list of other jobs upon which this job depends. Zuul will not start executing this job until all of its dependencies have completed successfully, and if one or more of them fail, this job will not be run.
allowed-projects
A list of Zuul projects which may use this job. By default, a job may be used by any other project known to Zuul, however, some jobs use resources or perform actions which are not appropriate for other projects. In these cases, a list of projects which are allowed to use this job may be supplied. If this list is not empty, then it must be an exhaustive list of all projects permitted to use the job. The current project (where the job is defined) is not automatically included, so if it should be able to run this job, then it must be explicitly listed. By default, all projects may use the job.
post-review
A boolean value which indicates whether this job may only be used in
pipelines where pipeline.post-review
is true
. This is
automatically set to true
if this job uses a secret
and is defined in a
untrusted-project
. It
may be explicitly set to obtain the same behavior for jobs defined in
config projects <config-project>
. Once this is
set to true
anywhere in the inheritance hierarchy for a
job, it will remain set for all child jobs and variants (it can not be
set to false
).
branches
A regular expression (or list of regular expressions) which describe on what branches a job should run (or in the case of variants, to alter the behavior of a job for a certain branch).
This attribute is not inherited in the usual manner. Instead, it is used to determine whether each variant on which it appears will be used when running the job.
If there is no job definition for a given job which matches the
branch of an item, then that job is not run for the item. Otherwise, all
of the job variants which match that branch are used when freezing the
job. However, if job.override-checkout
or job.required-projects.override-checkout
are set for a
project, Zuul will attempt to use the job variants which match the
values supplied in override-checkout
for jobs defined in
those projects. This can be used to run a job defined in one project on
another project without a matching branch.
This example illustrates a job called run-tests which uses a nodeset based on the current release of an operating system to perform its tests, except when testing changes to the stable/2.0 branch, in which case it uses an older release:
- job:
name: run-tests
nodeset: current-release
- job:
name: run-tests
branches: stable/2.0
nodeset: old-release
In some cases, Zuul uses an implied value for the branch specifier if none is supplied:
- For a job definition in a
config-project
, no implied branch specifier is used. If no branch specifier appears, the job applies to all branches. - In the case of an
untrusted-project
, if the project has only one branch, no implied branch specifier is applied tojob
definitions. If the project has more than one branch, the branch containing the job definition is used as an implied branch specifier.
This allows for the very simple and expected workflow where if a
project defines a job on the master
branch with no branch
specifier, and then creates a new branch based on master
,
any changes to that job definition within the new branch only affect
that branch, and likewise, changes to the master branch only affect
it.
See pragma.implied-branch-matchers
for how to override
this behavior on a per-file basis.
files
This indicates that the job should only run on changes where the specified files are modified. Unlike branches, this value is subject to inheritance and overriding, so only the final value is used to determine if the job should run. This is a regular expression or list of regular expressions.
irrelevant-files
This is a negative complement of files. It indicates
that the job should run unless all of the files changed match
this list. In other words, if the regular expression
docs/.*
is supplied, then this job will not run if the only
files changed are in the docs directory. A regular expression or list of
regular expressions.
Project
A project corresponds to a source code repository with which Zuul is
configured to interact. The main responsibility of the project
configuration item is to specify which jobs should run in which
pipelines for a given project. Within each project definition, a section
for each pipeline <pipeline>
may appear. This
project-pipeline definition is what determines how a project
participates in a pipeline.
Multiple project definitions may appear for the same project (for
example, in a central config projects <config-project>
as well as in
a repo's own .zuul.yaml
). In this case, all of the project
definitions for the relevant branch are combined (the jobs listed in all
of the matching definitions will be run). If a project definition
appears in a config-project
, it will apply to all branches of the
project. If it appears in a branch of an untrusted-project
it will only apply to changes on
that branch. In the case of an item which does not have a branch (for
example, a tag), all of the project definitions will be combined.
Consider the following project definition:
- project:
name: yoyodyne
check:
jobs:
- check-syntax
- unit-tests
gate:
queue: integrated
jobs:
- unit-tests
- integration-tests
The project has two project-pipeline stanzas, one for the
check
pipeline, and one for gate
. Each
specifies which jobs should run when a change for that project enters
the respective pipeline -- when a change enters check
, the
check-syntax
and unit-test
jobs are run.
Pipelines which use the dependent pipeline manager (e.g., the
gate
example shown earlier) maintain separate queues for
groups of projects. When Zuul serializes a set of changes which
represent future potential project states, it must know about all of the
projects within Zuul which may have an effect on the outcome of the jobs
it runs. If project A uses project B as a library,
then Zuul must be told about that relationship so that it knows to
serialize changes to A and B together, so that it does not merge a
change to B while it is testing a change to A.
Zuul could simply assume that all projects are related, or even infer
relationships by which projects a job indicates it uses, however, in a
large system that would become unwieldy very quickly, and unnecessarily
delay changes to unrelated projects. To allow for flexibility in the
construction of groups of related projects, the change queues used by
dependent pipeline managers are specified manually. To group two or more
related projects into a shared queue for a dependent pipeline, set the
queue
parameter to the same value for those projects.
The gate
project-pipeline definition above specifies
that this project participates in the integrated
shared
queue for that pipeline.
project
The following attributes may appear in a project:
name
The name of the project. If Zuul is configured with two or more
unique projects with the same name, the canonical hostname for the
project should be included (e.g., git.example.com/foo). This can also be a regex.
In this case the regex must start with ^
and match the full
project name following the same rule as name without regex. If not given
it is implicitly derived from the project where this is defined.
templates
A list of project-template
references; the project-pipeline
definitions of each Project Template will be applied to this project. If
more than one template includes jobs for a given pipeline, they will be
combined, as will any jobs specified in project-pipeline definitions on
the project itself.
default-branch
The name of a branch that Zuul should check out in jobs if no better
match is found. Typically Zuul will check out the branch which matches
the change under test, or if a job has specified an job.override-checkout
, it
will check that out. However, if there is no matching or override
branch, then Zuul will checkout the default branch.
Each project may only have one default-branch
therefore
Zuul will use the first value that it encounters for a given project
(regardless of in which branch the definition appears). It may not
appear in a project-template
definition.
merge-mode
The merge mode which is used by Git for this project. Be sure this matches what the remote system which performs merges (i.e., Gerrit or GitHub).
Each project may only have one merge-mode
therefore Zuul
will use the first value that it encounters for a given project
(regardless of in which branch the definition appears). It may not
appear in a project-template
definition.
It must be one of the following values:
merge
Uses the default git merge strategy (recursive).
merge-resolve
Uses the resolve git merge strategy. This is a very conservative merge strategy which most closely matches the behavior of Gerrit.
cherry-pick
Cherry-picks each change onto the branch rather than performing any merges.
<pipeline>
Each pipeline that the project participates in should have an entry in the project. The value for this key should be a dictionary with the following format:
jobs
A list of jobs that should be run when items for this project are
enqueued into the pipeline. Each item of this list may be a string, in
which case it is treated as a job name, or it may be a dictionary, in
which case it is treated as a job variant local to this project and
pipeline. In that case, the format of the dictionary is the same as the
top level job
definition. Any attributes set on the job here will override previous
versions of the job.
queue
If this pipeline is a dependent
<pipeline.manager.dependent>
pipeline, this specifies the
name of the shared queue this project is in. Any projects which interact
with each other in tests should be part of the same shared queue in
order to ensure that they don't merge changes which break the others.
This is a free-form string; just set the same value for each group of
projects.
Each pipeline for a project can only belong to one queue, therefore
Zuul will use the first value that it encounters. It need not appear in
the first instance of a project
stanza; it may appear in secondary instances
or even in a project-template
definition.
debug
If this is set to true, Zuul will include debugging information in reports it makes about items in the pipeline. This should not normally be set, but in situations were it is difficult to determine why Zuul did or did not run a certain job, the additional information this provides may help.
Project Template
A Project Template defines one or more project-pipeline definitions which can be re-used by multiple projects.
A Project Template uses the same syntax as a project
definition, however,
in the case of a template, the project.name
attribute does not refer to the name of
a project, but rather names the template so that it can be referenced in
a project
definition.
Because Project Templates may be used outside of the projects where
they are defined, they honor the implied branch pragmas <pragma>
(unlike Projects). The same heuristics described in job.branches
that determine
what implied branches a job
will receive apply to Project Templates (with the
exception that it is not possible to explicity set a branch matcher on a
Project Template).
Secret
A Secret is a collection of private data for use by one or more jobs. In order to maintain the security of the data, the values are usually encrypted, however, data which are not sensitive may be provided unencrypted as well for convenience.
A Secret may only be used by jobs defined within the same project. Note that they can be used by any branch of that project, so if a project's branches have different access controls, consider whether all branches of that project are equally trusted before using secrets.
To use a secret, a job
must specify the secret in job.secrets
. Secrets are
bound to the playbooks associated with the specific job definition where
they were declared. Additional pre or post playbooks which appear in
child jobs will not have access to the secrets, nor will playbooks which
override the main playbook (if any) of the job which declared the
secret. This protects against jobs in other repositories declaring a job
with a secret as a parent and then exposing that secret.
It is possible to use secrets for jobs defined in config
projects <config-project>
as well as untrusted projects
<untrusted-project>
, however their use differs slightly.
Because playbooks in a config project which use secrets run in the trusted execution context
where proposed changes are not used in executing jobs, it is safe for
those secrets to be used in all types of pipelines. However, because
playbooks defined in an untrusted project are run in the untrusted execution context
where proposed changes are used in job execution, it is dangerous to
allow those secrets to be used in pipelines which are used to execute
proposed but unreviewed changes. By default, pipelines are considered
pre-review and will refuse to run jobs
which have playbooks that use secrets in the untrusted execution context
to protect against someone proposing a change which exposes a secret. To
permit this (for instance, in a pipeline which only runs after code
review), the pipeline.post-review
attribute may be explicitly set
to true
.
In some cases, it may be desirable to prevent a job which is defined
in a config project from running in a pre-review pipeline (e.g., a job
used to publish an artifact). In these cases, the job.post-review
attribute
may be explicitly set to true
to indicate the job should
only run in post-review pipelines.
If a job with secrets is unsafe to be used by other projects, the allowed-projects job attribute can be used to restrict the projects which can invoke that job.
Secrets, like most configuration items, are unique within a tenant, though a secret may be defined on multiple branches of the same project as long as the contents are the same. This is to aid in branch maintenance, so that creating a new branch based on an existing branch will not immediately produce a configuration error.
secret
The following attributes must appear on a secret:
name
The name of the secret, used in a job
definition to request the secret.
data
A dictionary which will be added to the Ansible variables available
to the job. The values can either be plain text strings, or encrypted
values. See encryption
for more information.
Nodeset
A Nodeset is a named collection of nodes for use by a job. Jobs may specify what nodes they require individually, however, by defining groups of node types once and referring to them by name, job configuration may be simplified.
Nodesets, like most configuration items, are unique within a tenant, though a nodeset may be defined on multiple branches of the same project as long as the contents are the same. This is to aid in branch maintenance, so that creating a new branch based on an existing branch will not immediately produce a configuration error.
- nodeset:
name: nodeset1
nodes:
- name: controller
label: controller-label
- name: compute1
label: compute-label
- name:
- compute2
- web
label: compute-label
groups:
- name: ceph-osd
nodes:
- controller
- name: ceph-monitor
nodes:
- controller
- compute1
- compute2
- name: ceph-web
nodes:
- web
nodeset
A Nodeset requires two attributes:
name
The name of the Nodeset, to be referenced by a job
.
nodes
A list of node definitions, each of which has the following format:
name
The name of the node. This will appear in the Ansible inventory for the job.
This can also be as a list of strings. If so, then the list of hosts in the Ansible inventory will share a common ansible_host address.
label
The Nodepool label for the node. Zuul will request a node with this label.
groups
Additional groups can be defined which are accessible from the ansible playbooks.
name
The name of the group to be referenced by an ansible playbook.
nodes
The nodes that shall be part of the group. This is specified as a list of strings.
Semaphore
Semaphores can be used to restrict the number of certain jobs which are running at the same time. This may be useful for jobs which access shared or limited resources. A semaphore has a value which represents the maximum number of jobs which use that semaphore at the same time.
Semaphores, like most configuration items, are unique within a tenant, though a semaphore may be defined on multiple branches of the same project as long as the value is the same. This is to aid in branch maintenance, so that creating a new branch based on an existing branch will not immediately produce a configuration error.
Semaphores are never subject to dynamic reconfiguration. If the value of a semaphore is changed, it will take effect only when the change where it is updated is merged. However, Zuul will attempt to validate the configuration of semaphores in proposed updates, even if they aren't used.
An example usage of semaphores follows:
- semaphore:
name: semaphore-foo
max: 5
- semaphore:
name: semaphore-bar
max: 3
semaphore
The following attributes are available:
name
The name of the semaphore, referenced by jobs.
max
The maximum number of running jobs which can use this semaphore.
Pragma
The pragma item does not behave like the others. It can not be included or excluded from configuration loading by the administrator, and does not form part of the final configuration itself. It is used to alter how the configuration is processed while loading.
A pragma item only affects the current file. The same file in another branch of the same project will not be affected, nor any other files or any other projects. The effect is global within that file --pragma directives may not be set and then unset within the same file.
- pragma:
implied-branch-matchers: False
pragma
The pragma item currently supports the following attributes:
implied-branch-matchers
This is a boolean, which, if set, may be used to enable
(True
) or disable (False
) the addition of
implied branch matchers to job and project-template definitions.
Normally Zuul decides whether to add these based on heuristics described
in job.branches
. This
attribute overrides that behavior.
This can be useful if a project has multiple branches, yet the jobs defined in the master branch should apply to all branches.
Note that if a job contains an explicit branch matcher, it will be used regardless of the value supplied here.
implied-branches
This is a list of regular expressions, just as job.branches
, which may be
used to supply the value of the implied branch matcher for all jobs and
project-templates in a file.
This may be useful if two projects share jobs but have dissimilar branch names. If, for example, two projects have stable maintenance branches with dissimilar names, but both should use the same job variants, this directive may be used to indicate that all of the jobs defined in the stable branch of the first project may also be used for the stable branch of the other. For example:
- pragma:
implied-branches:
- stable/foo
- stable/bar
The above code, when added to the stable/foo
branch of a
project would indicate that the job variants described in that file
should not only be used for changes to stable/foo
, but also
on changes to stable/bar
, which may be in another
project.
Note that if a job contains an explicit branch matcher, it will be used regardless of the value supplied here.
Note also that the presence of implied-branches does not automatically set
implied-branch-matchers. Zuul will still
decide if implied branch matchers are warranted at all, using the
heuristics described in job.branches
, and only use the value supplied here if
that is the case. If you want to declare specific implied branches on,
for example, a config-project
project (which normally would not use
implied branches), you must set implied-branch-matchers as well.