Seems we missed these in our previous commit.
Change-Id: I630d02e45f0a17cd6f5886fcf9a971a8264ebf10
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
To avoid confusion with nodepool-launcher, we've decided to rename
zuul-launcher to zuul-executor.
Change-Id: I7d03cf0f0093400f4ba2e4beb1c92694224a3e8c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Here we now support 3 tox jobs; cover, linters and py27.
As you can see by looking at the code, we are not defining any
variables (vars) in .zuul.yaml. This means, we create 3 separate
playbooks (tox-cover.yaml, tox-py27, tox-linters.yaml) which then
contain the variables we need for our tox role.
Change-Id: I42f8855f44da51cb3d23f1825bd3258937543b2f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This hasn't generally been triggered recently, and the occasions
where it have caused major memory usage, to the point where we
had difficulty identifying the underlying issue.
Change-Id: I5bef896f5065b5676731e3ddd66da27746134249
It appears newer versions of GitPython have slowed considerably.
Cap GitPython until https://github.com/gitpython-developers/GitPython/issues/605
is resolved.
Change-Id: Ie6c8722e8b607bb50e77fbad59e18363616f7e0d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The zuul v3 architecture does not require that this be supported
as an internal special case. Remove it for now in the expectation
that we will implement similar functionality using the auth system
and Ansible roles.
Change-Id: I9c1e05a716ce8cfb5f66c4688db110c320d33b4b
This is no longer used. Instead, the configloader is reponsible
for validating each configuration stanza as it is parsed. The
error reporting happens as changes are proposed to in-repo
configuration files. Most of the good paths are tested by the
tests which utilize those features. There are a few tests which
deliberately test syntax errors, though they are not as
comprehensive. Because a nearly complete running Zuul system is
required in order to parse a configuration file now, it is not
practical to test configuration syntax errors in the way that was
previously done in the layoutvalidator; therefore the tests
are removed without a replacement.
Change-Id: I774dccbc2460099fc24815ed96f0de7532157d31
Adds a link to a storyboard search for low-hanging-fruit.
Updates the roadmap to reflect some tasks having been completed,
and one re-ordered (we were able to run jobs on zuul earlier than
expected).
Change-Id: Ibd56f61a0b7b5a48c048c2277e7d050dd8ebf9c6
These were perhaps a bit brief considering their importance.
Also, expand the docstring for RepoFiles.
Change-Id: Iafcc35f17ac90a5f5d63fcf05652e3a138fcdcea
Yaml parse errors come with nice pointers for the line and column
in the file where the error occurred. Provide the same for
Zuul configuration language syntax errors.
This adds a yaml.Mark object (which points to a file location) to
each top-level zuul configuration object. We could add it to every
dictionary we parse, but at the moment, that seems excessive.
This uses the same mechanism to add the already existing zuul
source context to the same objects, and retires the previous method.
The new error message looks like:
----
Zuul encountered a syntax error while parsing its configuration in the
repo org/project on branch master. The error was:
extra keys not allowed @ data['foo']
The error appears in a job stanza with the content:
{'foo': 'error', 'name': 'project-test2'}
in "org/project/.zuul.yaml@master", line 2, column 3:
- job:
^
----
Change-Id: Iea496a9dca6d4cef51854afc5d3e042b3b99411a
Rather than identifying a Nodepool request as originating from
'zuul', identify it with the hostname of the zuul scheduler. This
way the operator of a site with multiple zuuls can identify whence
requests originated.
Change-Id: I69aaca90d2bc3fac5fe55cdd9c57d724f6e103b4
* Refactor of layout-tags.yaml to single-tenant layout config dir.
* Build tags now added as ansible vars rather than env vars.
* Removes old layout config and custom add_tags function.
Change-Id: I1edfa505df7c9cc1a212f9483fe50ac088f1174e
Signed-off-by: Cullen Taylor <cullentaylor@outlook.com>
After working with this system for a while, this TODO does not
suggest to me anything that should be done, so convert it to a
NOTE documenting why it is interesting.
Change-Id: I8e50bc081f33a8efb3e9281c6460e4fb8e3bbd2e
This was mostly already done when we created 'ProjectConfig' and
'ProjectPipelineConfig' classes. Clean up the remaining leftover
code and comments.
Story: 2000786
Task: 3310
Change-Id: I8a0bfb2cd148c174bc7e0b5121f8c39ed1d4645b
The SourceContext is extended to support storing a path, and we
define our own YamlLoader to use that to include a complete
file location in the yaml error message, rather than "unicode string".
The resulting report looks like:
Zuul encountered a syntax error while parsing its configuration in the
repo org/project on branch master. The error was:
while parsing a block collection
in "org/project/.zuul.yaml@master", line 2, column 1:
- job:
^
expected <block end>, but found '?'
in "org/project/.zuul.yaml@master", line 3, column 1:
foo: error
^
Change-Id: I51cec01e45adaad5abeda9bf665cc0358b45486f
This is how gear lets us know that it has been stopped by another
thread. Since we weren't catching these exceptions explicitly,
they were polluting the logs on shutdown (especially in tests).
Catch them and return immediately from the daemon run methods.
Change-Id: I6f845566c759651e630d7f93c94dce975ad53d30
If a change with a syntax error is proposed to a config-repo, the
error detection and reporting recently added will not activate
because config-repo changes are not dynamically evaluated. However,
as soon as that change landed, the configuration would attempt to
reload and fail due to the error, potentially putting the operator
in a catch-22 that would require external intervention.
To avoid that, evaluate and construct a dynamic layout *including*
any changes to config-repos for the purpose of detecting configuration
errors. Then discard that layout and construct a new one *excluding*
config-repos (as we do now) for actual use when running jobs.
Change-Id: I803cb05274e79e91477c202b754a1640391e63a6
If Zuul encounters a syntax error when loading an in-repo layout,
report that as a failure on the change. Try to provide as much
readable information as possible to allow the user to diagnose the
error.
All top-level configuration objects are updated to support a hidden
source context variable to aid in creating a useful error message.
Item.areAllJobsComplete is updated so that it returns true in the
case of a merge failure or config error. In Zuulv2, we knew what
jobs we would run even in those cases, but in Zuulv3, we don't, so
the item does not have a job tree, so in those cases, the item does
not report that all jobs are complete. Returining True in this case
allows the NNFI algorithm to continue and expel the item from the
queue, avoiding an inifinite loop.
The merge failure accessor method is simplified.
Change-Id: I9cf19b87c6af5926b5e8bb403b81ce0470e3592d
This test was already refactored for v3, but it was causing occasional
timeouts. This is a very merge-heavy test, and as such, sometimes runs
out of time when contending for resources. So we'll bump the overall
hard-timeout by 30 seconds which should prevent us from timing out at
that level, and also raise the soft wait timeout to 90 seconds for just
this test.
Change-Id: I74e54d8a7d9b813772c498ccb87fb6644b533842
Paul Belanger