This will prevent the executor from taking too many jobs all at
once which could cause the system to become unstable. It adds a
config option to allow scaling based on observed capability of the
system since load average can be subject to factors outside of
Zuul's control. The default of 2.5 was selected based on highly
scientific observation of a single graph from OpenStack infra's
cacti that looks about right.
Change-Id: I346a92fe7185d7ef216cd303e19acaa59c6d6101
We set precedence in our pipeline configs but we do not pass it through
to the nodepool NodeRequest priority, which means that check can starve
gate.
Change-Id: Id3fa6f9ad6bdf23bf3af43c48289c4b918ea04f1
* When loading from multiple files, ensure that the correct filename
is included in the path (rather than just the last).
* Keep track of the line number where jobs are defined and include it
in the job representation. This is used in the inheritance_path
var to aid in debugging.
Change-Id: Ia3f0d03cd45d6c3b24b3dc7b663d1175cc0a18fc
This is temporary and undocumented -- I plan on reworking this into
structured data before finalizing the interface. Will add
documentation then.
Change-Id: I4b9bceddf13e384c18bb27be8c493318c5af0fb4
Even if things are not in an expected state (eg, node not locked)
or we encounter errors updating node state, always try to unlock
the nodes when returning a nodeset. Finally, if that too fails,
it will be logged but not treated as a fatal exception so that
we don't get stuck in queue processing.
Change-Id: Iab3bae4146c62067342e0c521a6289d045e65a5c
* Always send a build start event for noop jobs so that we get
start and end times for them
* Handle builds without start or end times in the sql reporter.
These should no longer include noop builds, but may still include
SKIPPED builds.
Test both.
Change-Id: I73eb6bda482ebb515d231492c0769d49cf6ff28a
Jobs that need this expect it as an environment variable.
Update the migration script in case it gets used again.
Change-Id: Ic4d372a795e17a33116d19e9e4603fb5a08fe152
This is something we had in Zuul v2.5 but slipped through in v3.
Executors delay getting new jobs from gearman in proportion to
the number of jobs they're already running, so as to avoid a single
executor ending up running all the jobs.
Change-Id: I75b87c09d507ee7c4785acf52e75e918bb0ab16f
In order to find project templates that need to be expanded from job
matchers that are not project specific, we need to make a pass through
all of the projects assuming all templates need to be expanded. We then
look at the result of the expansion to see if anything was actually
done. As part of this, we also collect same-expansions on a job basis to
track if a given job always has the same matcher expansion. If it does,
we can apply that to the job definition and not to the project-pipeline
defintion, which could lower the number of templates that need to be
expanded.
This may be the ugliest code I've ever written. I'm sorry.
Also fix a bash bug in the run-migration script that caused final to
always get run regardless of flag setting. Whoops.
Change-Id: I523909e5242e0db125b7560cbdcd9ac41ca6c72f
Add a boolean to indicate whether a project which appears in the
projects list is there because it was in required-projects or
because of the proposed change structure.
Change-Id: Ibfc2c82f863d88b2a65bd8990933146fc9ff3ee9
Run the Ansible 'setup' module on all hosts in the inventory
at the start of the job with a 60 second timeout. If we
aren't able to connect to all the hosts and gather facts
within that timeout, there is likely a network problem
between here and the hosts in the inventory; return them and
reschedule the job.
Change-Id: I121514aa1f3c9ac1a664322c8d7624703146f52f
We've told zuul about a new branch, so it's going to go look for
configuration in it, but while that's happening, we're adding a
new change upstream. Adding the change will alter the upstream
repo sufficiently that queries to the fake gerrit to ask about
refs are likely to be affected by the process of adding the change.
To correct, wait for the system to stabilize after adding the branch
and before adding the new change.
Change-Id: Ie3e2e9b5ad35ba99c32261ab8e9b6265a95b7cde
Return a message in the live log stream about failure to connect
to the executor providing the finger stream.
Change-Id: I6d111461096446550e541e26cf18486e6acbf189
These are the things one does after running the script. Update the job
to run the actual script we're running to generate the data the way
we're running it.
Change-Id: I62d75d561efbb290d2fccbabf4fabfbf705e6288
Turns out 99 isn't a good prefix.
Also, remove the move argument now that we're merging with the existing
projects.yaml.
Also, stop running zuul unittests on migration script changes. They are
not relevant.
Change-Id: I10ed8cae64c82ed5afd01bb03a74ffc4fd2d87ee
Clint Byrum
Monty Taylor
James E. Blair
Zuul
David Shrewsbury