We inadvertently fetch the PR object twice because of the way
we were fetching reviews. Keep it around so we can make one
less API call.
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
Change-Id: If5260278adb525566d99eedaecaf8b4f5077d43e
Maintain a cache of sha->PR which we update every time we fetch
a pull_request object from github. Use this in the status event
handler to map the status event to relevant PRs.
Change-Id: Ie811329429e1b672ce012767a475447f80832ee8
This change fixes the buildset route limit which incorrectly match
the number of build instead of the number of buildset.
Change-Id: Ia5775a32e646a7107a6dd04328ac36ec681b3b50
This moves event processing to its own class, so that it's
easier to bundle all of the data related to an event along with
an event-specific logger.
This logs the delivery ID for every line when we're preparing the
event. It also logs the start time and queue length as well as the
end time, even on error.
Change-Id: I941a74ecbdb418cf94537ca9f8f1917a5e38dd33
It's not obvious why we're renaming the attribute here. Leave a
clue for later explorers.
Also, make test_artifacts handle artifacts being returned in a
different order.
Change-Id: Iffa216105f7e4951ea57fc073dacaa4040f82040
A recent attempt to use the artifact return feature of zuul_return
exposed some rough edges. These two changes should make it much
easier to use.
First, return artifacts as a dictionary instead of a list. This
requires that they have unique names (which is no bad thing -- what
would two artifacts named "docs" mean anyway?). But mainly it allows
the dict merging behavior of zuul_return to be used, so that one
playbook may use zuul_return with some artifacts, and another playbook
may do the same, without either needing to load in the values of
the other first (assuming, of course, that they use different artifact
names).
Second, add a metadata field. In the database and API, this is JSON
serialized, but in zuul_return and zuul.artifacts, it is exploded into
separate fields. This lets jobs do things like associate versions or
tags with artifacts without having to abuse the url field.
Change-Id: I228687c1bd1c74ebc33b088ffd43f30c7309990d
Put the logging under the zuul hierarchy, and also add the ref
to the event representation. Both of these changes are in aid
of better visibility of git events.
Change-Id: Iac00be7536ad236e366cd793366ac6819357c17c
For some tasks the Ansible log will not contain enough information to
debug failures (e.g. missing role with include_role).
Ansible treats those issues not like an error (exit code 1) but like a
failed task, leading to an exit code of 2.
Change-Id: Iea754814e3d55be6be1c2de7f2d45ceda757f480
Like pre-run and post-run, allow a user to run a list of playbooks for
a job. One example would be your job workflow would be to run multiple
playbooks over using a site.yaml file with include_playbook commands.
A second use case, more related to job design. With multiple playbooks
support for job.run, the first playbook would be use deploy your server
and the second playbook to validate the server was provisioned properly.
Today, this can be done using a single run and post-run playbooks,
however if post-run fails, zuul will return POST_FAILURE, not FAILURE.
Not a large issue, but could be confusing to users when POST_FAILURE is
returned.
While it is possible a user could create a single site.yaml playbook,
and use multiple include_playbook statements to get this functionality,
there are downsides to this approach (mostly with the leaking of
variables). Today, operators simply run ansible-playbook multiple times
with the specific playbooks they only want.
Story: 2002543
Task: 22101
Change-Id: I6268d9944e745cc07407ea7dd040fbfeb79dad4d
Related-To: https://review.openstack.org/519596
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The Service Workers seem to be consistently causing issues for people
that are strange, meaning many of our deployers are disabling them.
Since they aren't super necessary for the Zuul use case, change the
default behavior to be to disable them instead of enable them.
Change-Id: Iea8348a3b007badaae74fc1837b55bb0b076ac65
So that people can re-use the Dockerfiles to build zuul images
but with different flags set, plumb the env vars through here
as ARG entries.
Also, fix 2 doc references that were misspelled.
Change-Id: I320a496eadf4132fc0583dd48a87024a2ff61a07
nodejs v10 is the latest LTS release of node. Update to use it
instead of v8. One would not expect this to impact us much, but
in theory v10 is faster so we could see quicker compile times.
Update zuul-build-dashboard to run on changes to
install-js-tools.
Downloaded the 10.x version of the upstream setup script and
updated the header.
Made a meaningless change in package.json to trigger build jobs.
Change-Id: Ibfc732a9b725f8ee5f0eb51ece11692694956592
We're seeing a lot of occurrences where a test case failed with the
message 'Timeout waiting for zuul to settle' withoutany obvious
reason. After digging into one of these logs more deeply it gets
aborted just while doing regular git stuff during job
preparation. Thus it looks like the default timeout of 30 seconds is
just not enough on some slower nodes.
[1] Log snippet
2019-02-04 06:39:57,848 zuul.AnsibleJob DEBUG [build: 19a6f8c3d35e49c29f89e83b8cb1204a] Create reference refs/heads/master at 1b86034fbdea80f4973c15b72ac85a676da0c315 in /tmp/tmpljq8cb9b
/zuul-test/19a6f8c3d35e49c29f89e83b8cb1204a/work/src/review.example.com/org/project
2019-02-04 06:39:57,859 git.cmd DEBUG Popen(['git', 'cat-file', '--batch-check'], cwd=/tmp/tmpljq8cb9b/zuul-test/19a6f8c3d35e49c29f89e83b8cb1204a/work/src/review.example.com/org/
project, universal_newlines=False, shell=None)
2019-02-04 06:39:57,868 zuul.test ERROR Timeout waiting for Zuul to settle
2019-02-04 06:39:57,869 zuul.test ERROR Queue status:
2019-02-04 06:39:57,869 zuul.test ERROR <queue.Queue object at 0x7fea0c2426d8>: True
2019-02-04 06:39:57,869 zuul.test ERROR <queue.Queue object at 0x7fea0c242278>: True
2019-02-04 06:39:57,869 zuul.test ERROR <zuul.lib.queue.MergedQueue object at 0x7fea0c242630>: True
2019-02-04 06:39:57,869 zuul.test ERROR <queue.Queue object at 0x7fea0c095eb8>: True
2019-02-04 06:39:57,869 zuul.test DEBUG <Build 19a6f8c3d35e49c29f89e83b8cb1204a of project-test1 voting:True on <Worker Unknown>> has not reported start
2019-02-04 06:39:57,869 zuul.test ERROR All builds waiting: False
2019-02-04 06:39:57,870 zuul.test ERROR All builds reported: True
2019-02-04 06:39:57,870 zuul.test ERROR All requests completed: True
2019-02-04 06:39:57,870 zuul.test ERROR Merge client jobs: set()
Change-Id: I48b0e452c894e0767625465daa749837bb1bd8fd
We use the SimpleHTTPRequestHandler as proxy in web urls test and as
fake gerrit server. By default it pollutes stderr with every request
it receives. This produces much visual noise when running tox. By
overriding the log_message method this can be logged properly.
Change-Id: Ifc5f1558dc374fa884fd5625221ba4089d533572
If there is already a post pipeline defined a force-merge of a change
referencing a non-existent project template currently can wedge the
scheduler. In this case an exception is raised during reporting which
throws the run handler completely out of its loop and the loop starts
over from the beginning. However the item to be reported is not
consumed in this case so zuul is stuck in an exception loop [1] and
can only be recovered by a restart.
This can be fixed by catching the exception and continuing the
reporting.
[1] Traceback:
2019-01-28 07:33:57,304 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.6/site-packages/zuul/scheduler.py", line 1033, in run
while (pipeline.manager.processQueue() and
File "/opt/zuul/lib/python3.6/site-packages/zuul/manager/__init__.py", line 768, in processQueue
item, nnfi)
File "/opt/zuul/lib/python3.6/site-packages/zuul/manager/__init__.py", line 735, in _processOneItem
self.reportItem(item)
File "/opt/zuul/lib/python3.6/site-packages/zuul/manager/__init__.py", line 880, in reportItem
item.reported = not self._reportItem(item)
File "/opt/zuul/lib/python3.6/site-packages/zuul/manager/__init__.py", line 920, in _reportItem
if not layout.getProjectPipelineConfig(item):
File "/opt/zuul/lib/python3.6/site-packages/zuul/model.py", line 3435, in getProjectPipelineConfig
templates = self.getProjectTemplates(template_name)
File "/opt/zuul/lib/python3.6/site-packages/zuul/model.py", line 3374, in getProjectTemplates
raise TemplateNotFoundError("Project template %s not found" % name)
zuul.model.TemplateNotFoundError: Project template foo not found
Change-Id: I1a3b59dadbd9337a8ba5b146f09ad093a0123ce8
When running test cases in the debugger it spits out resource warnings
about non-closed streams [1]. Explicitly closing the streams of the
subprocesses and log streamer fixes this warning. Maybe this even
solves the memory leak we're currently seeing in the executors.
[1] Trace:
/zuul/executor/server.py:1894: ResourceWarning: unclosed file <_io.BufferedReader name=60>
self.proc = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/zuul/executor/server.py:124: ResourceWarning: unclosed file <_io.BufferedReader name=11>
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Change-Id: I65f191dc2e50f9c84f5bf6a3967d768d7ebe6b04
This change improves the status reducer to clear the status for the
tenant set action. This prevent incorrect status and flickering to
occur when changing tenant.
Change-Id: Ie910e83eba264a6668fa5fce7ebf71fe6c8cc36b
When dealing with large repos or slow connections to the scm the
default clone timeout of 5 minutes may not be sufficient. Thus a
configurable clone/fetch timeout can make it possible to handle those
repos.
Change-Id: I0711895806b7cbcc8b9fa3ba085bcf79d7fb6665
When adding a change to a pipeline we have two filter steps before
getting the change queue and doing further checks. The first one calls
isChangeReadyToBeEnqueued which is used by the dependent pipeline
manager to check if a change can merge. This causes potentially
various api calls as this updates the change, analyzes status checks
etc and thus is potentially expensive. The check for pipeline
requirements is purely local and thus cheap. In many cases the
pipeline requirements could filter out changes before the can merge
check.
Switching these two filters can save us some api calls and improve the
performance of the scheduler run_handler loop.
Change-Id: I3d3ea186e7852ae9c253d29f78dd5d7df81e71aa
Adds support for expressing artifact dependencies between jobs
which may run in different projects.
Change-Id: If8cce8750d296d607841800e4bbf688a24c40e08
James E. Blair
Zuul
Quique Llorente
Tristan Cacqueray
Monty Taylor
Simon Westphahl
Paul Belanger
Tobias Henkel