Add a HOWTO that walks through setting up a Matrix account with
Element and matrix.org, joining #zuul, and some optional additional
steps to save encryption keys and register with a 3pid provider.
Change-Id: I217c44a7270033998113e434f40a49c2dbb6aaef
This describes the reasoning and process for moving from IRC on OFTC
to Matrix.
If we obtain consensus on this spec, that indicates that as a community
we desire the outcome described in this spec. The next step would then
be to create an infra-spec to obtain consensus that OpenDev is willing
to help achieve that and the technical details for doing so.
To be clear, the Zuul community doesn't get to decide what OpenDev will
do. Instead we're going to decide what we want to ask of OpenDev, and
what we're willing to volunteer to help accomplish.
Change-Id: I3b4c0f9e3aa8e5b1464f196c124c38bd0ee0a480
This adds a specification describing how resource limits can be defined
and enforced on a per-tenant level via Nodepool.
Change-Id: I799d80b60b8b6eaf14ac284fa74574e535b9c58c
We have an example of the re-run trigger in the github check pipeline.
Add it also for the gate pipeline.
Change-Id: I248cb7d46cf05a50226b86bbb0f0cc066cc8f97e
This allows jobs to request multiple semaphores. Zuul will wait until
all are available before acquiring them* and will not start a job unless
all have been acquired.
This is useful for jobs which require access to mulitple limited resources
(especially if other jobs require access to a subset or superset of those
same resources).
* Implementation note: for efficiency, we actually do acquire them
one-by-one but then release any which have been acquired if they are
not all available. This all happens very quickly within a single
attempt to start a job. We don't hold semaphores while we wait for
others as that could cause deadlocks.
To be consistent with other job attributes which accept lists, this
deprecates job.semaphore and replaces it with job.semaphores.
Change-Id: I295a891a2d02b904820d8f60afe8ef862693b75d
In order to expose the length of the connection event queues we need to
introduce `getEventQueue()` as a new interface in the connection base
class.
The queue length will be available in the status JSON output for display
in zuul-web as well as a statsd gauge metric:
zuul.scheduler.eventqueues.connection.<connection-name>
This gives an operator more insight into possible bottlenecks during
event pre-processing.
Change-Id: Ie2e09e0c7cd166dce5b884c3e3d12ead42b05397
Expose the length of the pipeline event queues as statsd metrics.
The new gauge metrics are:
* zuul.tenant.<tenant>.pipeline.<pipeline>.trigger_events
* zuul.tenant.<tenant>.pipeline.<pipeline>.result_events
* zuul.tenant.<tenant>.pipeline.<pipeline>.management_events
Change-Id: Ie2e3bfd8be58fa60418fad0f4cd79dd095e7caba
The dequeue reporting will ensure that zuul marks check runs as
cancelled if jobs have been aborted.
Change-Id: I710c0e3544934179449a098dd3b0ae3fc6e94465
Drivers should pass event data unmodified so trigger documentation can
refer directly to upstream api documentation like GitHub. This is
handled incorrectly for the check_run action 'rerequested' which had
been simplified to 'requested'. Fix this by removing the translation
and add a temporary compatibility layer to prepare future removal.
Change-Id: I2a75c92d1aeacf261b517a328d91ea02f6013762
Defining the queue on the pipeline is deprecated so switch the example
snippet to define the queue on the project.
Change-Id: I2d4dfaf41dc6b7c4556c0859a531fb18905d2103
This adds the concept of a 'scheme' to the merger. Up to this point,
the merger has used the 'golang' scheme in all cases. However it is
possible with Gerrit to create a set of git repositories which collide
with each other using that scheme:
root/example.com/component
root/example.com/component/subcomponent
The users which brought this to our attention intend to use their repos
in a flat layout, like:
root/component
root/subcomponent
To resolve this we need to do two things: avoid collisions in all cases
in the internal git repo caches of the mergers and executors, and give
users options to resolve collisions in workspace checkouts.
In this change, mergers are updated to support three schemes:
* golang (the current behavior)
* flat (new behavior described above)
* unique
The unique scheme is not intended to be user-visible. It produces a
truly unique and non-conflicting name by using urllib.quote_plus. It
sacrifices legibility in order to obtain uniqueness.
The mergers and executors are updated to use the unique scheme in their
internal repo caches.
A new job attribute, 'workspace-scheme' is added to allow the user to
select between 'golang' and 'flat' when Zuul prepares the repos for
checkout.
There is one more kind of repo that Zuul prepares: the playbook repo.
Each project that supplies a playbook to a job gets a copy of its repo
checked out into a dedicated directory (with no sibling repos). In that
case there is no risk of collision, and so we retain the current behavior
of using the golang scheme for these checkouts. This allows the playbook
paths to continue to be self-explanatory. For example:
trusted/project_0/example.com/org/project/playbooks/run.yaml
Documentation and a release note are added as well.
Change-Id: I3fa1fd3c04626bfb7159aefce0f4dcb10bbaf5d9
The MQTT reporter now includes artifacts for completed builds. Systems
which watch for MQTT events can now directly consume those artifacts without
the intermediate step of looking them up via the API.
Change-Id: I9df9e1dfd6854518c110dd65d4f89dea449c6fc0
This change adds a new prometheus_port option to start a metric server
to be scrapped by a prometheus service. By default, the server exposes
process informations.
Change-Id: Ie329df6adc69768dfdb158d00283161f8b70f07a
This is likely to be needed by executors as well since passing
decrypted secrets to the executors via zookeeper has the same
encrypted-at-rest concerns as they keystore itself. To avoid
confusion around executors needing a zuul.conf with a scheduler
section, start a new keystore section which we can later indicate
is used by schedulers and executors. It also makes it convenient
to add new options (like those dealing with rotation, or even
using an external keystore).
Also change some log levels from debug to info where it's useful
for the operator to know that the backup keystore was used (or
a key was generated).
Change-Id: If2491bbe4eb80b76435a274cf5354a4918315e65
As a preparation for the HA scheduler, project secrets keys and SSH keys
will now also be stored in Zookeeper. All private data in Zookeeper will
be encrypted at rest.
Existing keys on the filesystem will be automatically imported into
Zookeeper and new keys will still be available as files for backup.
Change-Id: I2a7d1a555f1db1f2178d3bb2f06756ecc8bc7a81
Zookeeper docker 'latest' image was updated from 3.6.2 to 3.7.0
Some exceptions java.nio.file.NoSuchFileException about files
- /var/certs/keystores/examples_zk_1.examples_default.pem
- /var/certs/certs/cacert.pem
appeared.
This change adds a check on the last file created by tools/zk-ca.sh
before running zookeeper service.
Change-Id: I15b67977a8b14bb83474390786ab47000e7be07c
When running large distributed deployments it can be desirable to keep
traffic as local as possible. To facilitate this zuul supports zoning
of zuul-executors. Using zones executors only process jobs on nodes
that are running in the same zone. This works well in many
cases. However there is currently a limitation around live log
streaming that makes it impossible to use this feature in certain
environments.
Live log streaming via zuul-web or zuul-fingergw requires that each
executor is directly addressable from zuul-web or zuul-fingergw. This
is not the case if
* zuul-executors are behind a NAT. In this case one would need to
create a NAT rule per executor on different ports which can become a
maintenance nightmare.
* zuul-executors run in a different Kubernetes or OpenShift. In this
case one would need a Ingress/Route or NodePort per executor which
also makes maintenance really hard.
Change-Id: Iecff1b05a93b34c2a80d18df21eafdcc46f6fdfa
This updates the new metric added in
I453ff6dfe479a48a32f4f581f5d07f4ee6b4d804 to:
* Rename it event_enqueue_processing_time to make it more clear
what is being measured.
* Move it out of the pipeline hierarchy since it's not pipeline-
specific.
* Add a companion event_enqueue_time which also includes the
driver pre-processing time.
* Rename the internal variable from "trigger_timestamp" to
"arrived_at_scheduler_timestamp" because we have a lot of timestamps
and they are confusing and the extra verbosity helps keep them
straight.
* Switches from monotonic to time.time() to match what the drivers
are setting for the initial event timestamp.
Change-Id: Ie283c949bd5f4e133152050fa9c027d4882a8c0e
At present, if we read components page[1] of Zuul doc, database
seems to be required only when SQL connection is specified in zuul.conf.
However, database is required even if SQL connection is not specified
in zuul.conf.
To avoid this confusion, this commit modifies components page of doc.
[1] https://zuul-ci.org/docs/zuul/discussion/components.html
Story: 2008725
Task: 42069
Change-Id: Id1d68bd5cff43e9479dab851018799691c8e3f57
zuul.artifacts example fails to be parsed with an Ansible parse error:
ERROR! Syntax Error while loading YAML. mapping values are not
allowed in this context
Change-Id: I7d344e96de74c3b55ee7430ae391d344b2485c2d
When running a deployment using the zone feature it can be troublesome
to run nodeless jobs. The reason for this is that zoned executors
don't run jobs that have no zone at all. This can be solved by either
running an additional set of unzoned executors or with this change
allow some executors also to process unzoned jobs. As running
additional unzoned executors just for nodeless jobs can increase
operations overhead this optional setting can be very useful for those
setups.
Change-Id: I9c025cdbe17a55f47f0cbd97b8b593727fc97e2d
This gives us 3 sets of executor stats: zoned, unzoned, and all.
The zoned and unzoned stats are moved to a hierarchy that avoids
name collisions.
The only way to detect whether an executor in a zone is online
is with a function with its zone name registered. Since the only
function currently registered with a zone name is "execute:execute"
and that is unregistered when an executor is online but not accepting,
we need to add a new dummy function in order to do this accounting.
This also updates the docs and adds a release note about the (minor)
change in stats meaning.
Change-Id: Ie28963426024f2d54275426794549f31ace9d998
Change Ic121b2d8d057a7dc4448ae70045853347f265c6c omitted support
for enqueuing changes behind a cycle in order to avoid having them
show up in the middle of the cycle. However, we can process changes
behind a cycle if we wait until we have completely enqueued the
cycle before we embark on enqueing changes behind it.
This also removes unused pipelines which cause extra processing and
output in the unit tests.
Also add a release note about circular deps, and describe additional
caveats about using circular dependencies in the documentation.
Change-Id: I64c0d4e8c20e4638bbafb18409cd28c062369738
Allow Zuul to process circular dependencies between changes. Gating of
circular dependencies must be explicitly enabled on a per tenant or
project basis.
In case Zuul detects a dependency cycle it will make sure that every
change also include all other changes that are part of the cycle. However
each change will still be a normal item in the queue with its own jobs.
When it comes to reporting, all items in the cycle are treated as one
unit that determines the success/failure of those changes.
Changes with cross-repo circular dependencies are required to share the
same change queue.
Depends-On: https://review.opendev.org/#/c/643309/
Change-Id: Ic121b2d8d057a7dc4448ae70045853347f265c6c
This adds documentation for zuul-executor graceful. In the process we
tweak the docs for pause as well.
Change-Id: Ieec5740f9a3d1857036d5d6e2af7cb30f9d77f94
This change updates the config documentation to indicate that
the config project branch can be configured.
Change-Id: Ib7e77ac289b2bc97b6c851744f0999773e091951
Part of point 5 in https://etherpad.openstack.org/p/zuulv4
Connection is idle for now.
Also update component documentation.
Change-Id: I97a97f61940fab2a555c3651e78fa7a929e8ebfb
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".
Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
On the way towards a fully scale out scheduler we need to move the
times database from the local filesystem into the SQL
database. Therefore we need to make at least one SQL connection
mandatory.
SQL reporters are required (an implied sql reporter is added to
every pipeline, explicit sql reporters ignored)
Change-Id: I30723f9b320b9f2937cc1d7ff3267519161bc380
Depends-On: https://review.opendev.org/621479
Story: 2007192
Task: 38329
Referring to a queue makes more sense in the project where it has more
use such as allowing circular dependencies later. This makes it
possible to refer to the queue on project level which takes precedence
over referring queues in the pipeline config. We'll also deprecate
referring to the queue in the pipeline config and remove this in a
later release.
Change-Id: I14533ae57cfe688b55c26a1e17ab38e133180b28
We have several large projects with most of the time long gate
queues. Those projects typically work on master and few release
branches where the changes in the release branches are more important
to the changes for master. Currently all of those changes are queued
up in a shared gate queue which makes the process of getting changes
into the release branches very slow especially if occasional gate
resets are involved. In order to improve this allow specifying the
change queues per branch so we can queue up the changes for each
release branch in a separate queue.
This is done by adding a new config element 'queue' which can be
configured to work on a per branch level.
Change-Id: Ie5c1a2b8f413fd595dbaaeba67251da14c6b4b36
The timestamp field helps to prepare visualization in
Kibana.
The duration field was set to integer in Kibana object, but
the value was string, so the Kibana was doing incorrect visualizations.
Also this commit provides fix for rst block in elasticserarch driver
document.
Change-Id: I92a034d78f9193476eccecc7efb4a818d4b4a658
It has the capability to index build and buildset results.
With the help of tools like Kibana, advanced analytics
dashboard could be built on top of the Zuul Elasticsearch
index.
Optionally job's variables and zuul_return data can be
exported along with build results under the job_vars and
job_returned_vars fields.
Change-Id: I5315483c55c10de63a3cd995ef681d0b64b98513
-quick-start steps are modified and fit more to what a reader would do
-quick-start test code is mainly splitted into 2 files, one which is a
setup part as a role, the second one starts with cloning the test
repository, just like all followings tutorial will do
-some elementary steps when manipulating or checking gerrit are being
added as roles
tutorial ssh config: test ssh configuration has been modified to allow
using a known_hosts file for both someone executing localtest and
opendev.org's zuul. A reader executing the tutorial would still have to
accept the fingerprint. To do so, commit-msg hook is fetched manually,
otherwise it would be downloaded by git-review throught scp. Alas,
git-review doesn't allow to pass options to scp to provide a new
known_hosts file.
User's ssh key is used if ~/.ssh/id_rsa.pub is available, otherwise use
a generated one.
- "to_json | from_json | json_query" in test is due to an issue between
ansible and jmespath [1]
[1] https://github.com/ansible-collections/community.general/issues/320
Change-Id: Id5c669537ff5afc7468352139980ebade167d534
James E. Blair
Benjamin Schanzel
Artem Goncharov
Simon Westphahl
Albin Vass
Zane Bitter
Albin Vass
Tobias Henkel
melanie witt
Tristan Cacqueray
Guillaume Chauvel
vanou
Vitaliy Lotorev
Clark Boylan
Jan Kubovy
Pierre-Louis Bonicoli
Daniel Pawlik
Fabien Boucher