Job variants generally should not set the parent attribute. Instead,
the inheritance path of the reference definition of the job should be
used.
I had previously considered honoring parent in variants a harmless
undocumented feature (which permitted a sort of crude
multiple-inheritance). However, this makes it very difficult, if not
impossible, to create correct branch variants of jobs from the initial
branch point -- a new branch will end up with an identical copy of the
jobs from the master branch, including the parent attribute. Since
the behavior difference between the jobs on the two branches runs counter
to what the user would expect, let's ignore the parent attribute on
variants.
Change-Id: I85ffb014c73e39631c76debb04d8c9775ab097ad
The automatic statsd configuration based on env variables has
proven cumbersome and counter-intuitive. Move its configuration
into zuul.conf in preparation for other components emitting stats.
Change-Id: I3f6b5010d31c05e295f3d70925cac8460d334283
If the final post playbook fails, something has gone wrong with log
uploading, which means it's very hard to debug. Grab the contents of the
json log file, extract the log for the last playbook and add it to the
executor log.
Change-Id: Ia930311e121c350e73e41b20e9b742b2eac9c9f6
Timeout remote git operations after 300 seconds.
Because it could be in an invalid state, delete the local repo if a
timeout occurs (subsequent operations will recreate it).
This replaces our use of the clone_from() and fetch() methods from
GitPython with lower-level equivalents. The high-level methods
do not currently permit the hard timeout.
The GitPython requirement is changed to a temporary fork until both
https://github.com/gitpython-developers/GitPython/pull/682
and
https://github.com/gitpython-developers/GitPython/pull/686
end up in a release.
Change-Id: I7f680472a8d67ff2dbe7956a8585fb3714119e65
In project and project-template definitions, the existing voluptuous
schema for the jobs in the job list was vs.Any(str, dict). The contents
of the dict itself need to be validated though, the job being a dict
that looks like:
check:
jobs:
- project-test1:
- required-projects:
org/project2
Is invalid as the contents of the build-openstack-sphinx-docs job dict
should themselves be a string or a dict rather than a list. This updates
the error to be:
Zuul encountered a syntax error while parsing its configuration in the
repo org/project on branch master. The error was:
expected str for dictionary value @ data['check']['jobs'][0]['project-test1']
The error appears in the following project stanza:
project:
name: org/project1
check:
jobs:
- project-test1:
- required-projects:
org/project2
in "org/project/.zuul.yaml@master", line 4, column 3
The error, 'expected str for dictionary value' could probably be
improved at some point, but this is at least an error with a message
which is way better than 'Unknown configuration error'.
Split out the attributes of the job in the JobParser voluptuous schema
that can be used in job lists from the ones that can't. For now it's
only name that can't be used.
Also fix a test fixture that had a trailing : in it.
Change-Id: I217eb5d6befbed51b220d47afa18997a87982389
Together, these changes build an OpenStack-sized configuration in
8% of the time it currently takes.
Change-Id: I85f538a7ebdb82724559203e2c5d5380c07f07e7
Based on Jim's feedback, change the branch matching logic to always
have priority over ref matching. And v3 will always have refs, so no
need to check if that attribute exists. Also adds a test that checks the
current breakage of branch matching logic.
Change-Id: Iba148b73a77b3300ad84db1c05c083d2c82cd950
We parse the project-pipeline definition of a job at the location
of the project-pipeline. This includes both 'project' stanzas and
'project-templates' which are parsed in exactly the same way. This
normally gives us the behavior we expect in that the job variants
defined by the project or project-template appear to be defined in
the location of the project or project-template. However, in one
case, we want a 'late-binding' rather than 'early-binding' behavior.
When it comes to calculating implied branch matchers, we want to
use the value that would be derived if there were no project-template,
and instead the job were simply defined on the project stanza itself.
What is intended to happen is that project-pipeline job variants in
a config project should never have implied branch matchers (since
config projects don't have more than one branch). However, project-
pipeline job variants on in-repo project stazas should get an implied
branch matcher for the branch it's defined on. This is how we end up
with behavior where the project definition in a project's master branch
controls behavior only on the master branch (unless branches are
explicitly specified), and the definition in a stable branch controls
only the stable branch.
That behavior should happen regardless of where a project-template is
defined. Currently we are setting an implied branch matcher for job
variants in a project template at the location of definition. Instead,
set them when the job is actually used in a project.
Change-Id: I5c8fbb3e0a2ecfac8bd95795be002e8cd15e61db
We set precedence in our pipeline configs but we do not pass it through
to the nodepool NodeRequest priority, which means that check can starve
gate.
Change-Id: Id3fa6f9ad6bdf23bf3af43c48289c4b918ea04f1
* Always send a build start event for noop jobs so that we get
start and end times for them
* Handle builds without start or end times in the sql reporter.
These should no longer include noop builds, but may still include
SKIPPED builds.
Test both.
Change-Id: I73eb6bda482ebb515d231492c0769d49cf6ff28a
We intended to have Nodesets be convenience methods for the 'nodes'
attribute of jobs, but be identical. When nodesets grew groups,
however, job.nodes did not. Because of the additional structure
that nodesets contain (to support groups, and likely vars in the
future), we can't simply extend the existing nodes parameter.
Add a new parameter, nodeset, which expects either a string or
an embedded nodeset definition. We're using the name 'nodeset'
here because 'nodes: nodes:' is difficult to understand.
Job.nodes will be removed soon.
(Re-proposed from I714887625c41bd1220ff05cd7356fbac589389c9)
Change-Id: I6c1c1e864704ac659efae9b28b140d9b37cef9d2
When a change that altered a dependent pipeline failed and was removed
from that pipeline, changes behind it did not have their configuration
reset; instead, the alteration introduced by the failed change lingered.
This was largely because the way we tested that a project participated
in a pipeline was incorrect -- we only checked that it was in the layout,
not the pipeline. That is corrected.
Additionally, the job graph for the second change was not reset. That's
because it was only created once, and was not cleared out when the
change's buildset is reset. That is corrected as well (though a future
enhancement would be to move item.job_graph into the BuildSet class to
better reflect its lifecycle).
Change-Id: Icdbc17bb21887a96f4118bb1a49ed7b00e1304e6
The file module is a normal module so goes through the normal action
plugin. This should validate that the normal action plugin will let a
good path through and block a bad path.
It should also show that the local override check doesn't fail closed,
as it should also allow this to happen.
Change-Id: I5611df4f0448685f2aa509c7d1b5f755f5610c9f
We greylist some modules in our action plugin blocking allowing them to
execute local code as long as it falls within safe constraints. Due to
the way ansible module loading works, a user could attack this by
creating a module in a local role or adjacent to a playbook that has the
same name as one of the modules we allow limited local execution. If
they did that it would allow them to execute arbitrary python code on
the executor.
Find the path of the module that will be executed in these cases and if
it is not within the ansible.modules package, disallow it. There are no
circumstances in which this is ok.
Change-Id: I7499e6b1091d745984ca36179de2793827c9f98f
This also fixes a condition in zuul_stream which would cause us
not to see stdout from tasks run on the executor (localhost).
Change-Id: I24df40adaf06a7bf0b238be33b8ab29cb692f836
There are some cases, such as the artifact upload job, where the job can
take a dict parameter and where it could be advantageous to allow other
people to re-use the job but passing in their own local secret data by
supplying variables to a variant. However, currently secrets carry with
them a name, which is used as the variable name in ansible.
Make a secret in a job config be able to be given as a string or a
dict. In the dict case, the name of the secret and the name it should be
added to ansible as are required. This allows someone to have a named
secret but to pass it to a job under a different name.
Change-Id: I27a82c6ee1cf7399353509f98a0a52536ebbc19a
This executes a job which writes a secret into a file in the jobdir,
which is typical of how we would expect many jobs which use secrets
to operate.
It also executes a similar job where ansible fails to write the file,
to test that error-handling code doesn't helpfully leak the secret.
It runs both of those tests with and without '-vvv' set.
It then searches for that secret in all files in the jobdir and
ensures it doesn't show up in any unexpected files. This includes
the ansible log(s).
Change-Id: Ie6ebe301f256d20e482b5f6c64f3ce2fb2b5135d
Porting this test needed some fundamental help because we haven't had
any tests that required changing the tenant config before.
Change-Id: Ife0bf391340d24f33b516afdb24626415ad7d65f
Story: 2001134
Task: 4847
Since jobs which use secrets in the trusted execution context are
always allowed, the name of this attribute was confusing. By renaming
it to 'post-review' (and the corresponding job attribute to
'post-review') we indicate what the actual concern is.
Change-Id: I59607621d5b99508b94074133bfc67e64e708a7d
It is possible we want to know the name of the cloud, which could be
different from nodepool.provider. In the case of openstack-infra, this
is to fix a DNS issue by when creating the mirror name of our regional
mirrors.
Change-Id: I3ac65744356e3fa25d10208d11be95dc16b1e2e7
Depends-On: Idc7686167d131d8e74d55b8f7f50224a1b782091
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
For ease of use in writing ad-hoc jobs, include a variable for the
directory on the nodes where the source code will be.
Change-Id: Idb7b3a14766497d7013e054fbe1279cc8809fdb9
Exposing the final job attribute make it possible to directly
configure a job as final.
Prohibit inheritance with final
This is no longer automatically set based on auth inheritance, so
now only exists as an attribute for a user to set explicitly.
The word "final" has a pretty specifig meaning for software developers
at least, so let's err on the side of safety there to provide folks
with the least surprise.
Also document it.
Change-Id: Ibeb7fd0ec1ce4f053a16066ccc8c2dd93c6f659e
Co-Authored-By: James E. Blair <jeblair@redhat.com>
Secrets are proving less useful than originally hoped because they
can not be effectively used in any jobs with untrusted children.
This change binds the secrets to the playbooks which use them, so
that child jobs are unable to access the secrets. This allows us
to create jobs with pre/post playbooks which use secrets which
are suitable for other jobs to inherit from.
Change-Id: I67dd12563f3abd242d6356675afed1de0cb144cf
The rtfd hook job just does an empty POST to a URI. There's no need to
allocate a node for that, we can just make REST calls from the executor.
Also, there is enough going on here that it needs to be documented. Add
a documentation section to the developer docs about what we're doing
with our ansible plugins. In support of that, add a simple sphinx domain
for ansible to allow us to easily link to upstream ansible documentation for
modules.
Change-Id: I9b0be1018388db7361aec10f30a70437de555615
This is not complete, but it's a start.
While doing this it became clear that the upstream csvfile lookup plugin does
not work in python3. Work around that by putting in a simpler version of
the code into our copy.
Change-Id: Ic84d8265e6fd7e15a0e5d66c781409a087d761d7
The FakeGithubConnection redefines the methods getCommitStatuses and
setCommitStatus for test purposes. This implies that the original
methods of GithubConnections are untested. This is an attempt to
remove these from FakeGithubConnection and pushing the test handling
into a FakeGithub object. This way we can do the test handling and at
the same time use the original methods of GithubConnection.
This also uncovered some test fixtures which are invalid in Github
context as there projects require the form <owner>/<project> which is
not matched by the 'common-config' which was used in many fixtures.
Change-Id: Ib3badca63b77166c1d69332121d78ef05bd899fe
