Docs: Zuul v2 is dead
Update the Zuul v3 page with correct links and description of promote pipeline and remove the now obsolete note - and follow it with renaming the document to zuul. Change-Id: I9c89cb56d4a318f3a234e7f2f08dabb46d0dfab6
This commit is contained in:
Andreas Jaeger
parent
f3def9b84a
commit
a6aaaed6a3
@ -14,7 +14,6 @@ Major Systems
|
||||
grafana
|
||||
grafyaml
|
||||
zuul
|
||||
zuulv3
|
||||
logstash
|
||||
elastic-recheck
|
||||
devstack-gate
|
||||
|
||||
@ -6,36 +6,69 @@ Zuul
|
||||
####
|
||||
|
||||
Zuul is a pipeline-oriented project gating system. It facilitates
|
||||
running tests and automated tasks in response to Gerrit events.
|
||||
running tests and automated tasks in response to Code Review events.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* https://zuul.opendev.org
|
||||
* zuul*.openstack.org
|
||||
* ze*.openstack.org
|
||||
* zm*.openstack.org
|
||||
:Puppet:
|
||||
* https://opendev.org/opendev/puppet-zuul
|
||||
* :git_file:`modules/openstack_project/manifests/zuul_prod.pp`
|
||||
* :git_file:`modules/openstack_project/manifests/zuul_dev.pp`
|
||||
* https://opendev.org/opendev/puppet-openstackci/src/branch/master/manifests/zuul_merger.pp
|
||||
* https://opendev.org/opendev/puppet-openstackci/src/branch/master/manifests/zuul_scheduler.pp
|
||||
:Configuration:
|
||||
* :config:`zuul/`
|
||||
* :config:`zuul.d/`
|
||||
* :config:`zuul/main.yaml`
|
||||
* :config:`zuul.d`
|
||||
:Projects:
|
||||
* https://opendev.org/zuul/zuul
|
||||
:Bugs:
|
||||
* https://storyboard.openstack.org/#!/project/zuul/zuul
|
||||
:Resources:
|
||||
* `Zuul Reference Manual <https://zuul-ci.org/docs/zuul>`_
|
||||
* `Zuul Reference Manual`_
|
||||
:Chat:
|
||||
* #zuul on freenode
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
The OpenStack project uses a number of pipelines in Zuul, as defined
|
||||
in :config:`zuul.d/pipelines.yaml`.
|
||||
The OpenStack project uses a number of pipelines in Zuul:
|
||||
|
||||
**check**
|
||||
Newly uploaded patchsets enter this pipeline to receive an initial
|
||||
+/-1 Verified vote.
|
||||
|
||||
**gate**
|
||||
Changes that have been approved by core reviewers are enqueued in
|
||||
order in this pipeline, and if they pass tests, will be merged.
|
||||
|
||||
**post**
|
||||
This pipeline runs jobs that operate after each change is merged.
|
||||
|
||||
**pre-release**
|
||||
This pipeline runs jobs on projects in response to pre-release tags.
|
||||
|
||||
**release**
|
||||
When a commit is tagged as a release, this pipeline runs jobs that
|
||||
publish archives and documentation.
|
||||
|
||||
**silent**
|
||||
This pipeline is used for silently testing new jobs.
|
||||
|
||||
**experimental**
|
||||
This pipeline is used for on-demand testing of new jobs.
|
||||
|
||||
**periodic**
|
||||
This pipeline has jobs triggered on a timer for e.g. testing for
|
||||
environmental changes daily.
|
||||
|
||||
**promote**
|
||||
This pipeline runs jobs that operate after each change is merged
|
||||
in order to promote artifacts generated in the gate
|
||||
pipeline.
|
||||
|
||||
Zuul watches events in Gerrit (using the Gerrit "stream-events"
|
||||
command) and matches those events to the pipelines above. If a match
|
||||
@ -52,10 +85,9 @@ each commit is correctly tested.
|
||||
Zuul's current status may be viewed at
|
||||
`<https://zuul.opendev.org/>`_.
|
||||
|
||||
Zuul's configuration is distributed across projects listed in
|
||||
:config:`zuul/main.yaml`. Anyone may propose a change to the
|
||||
configuration by editing configuration in those projects and submitting
|
||||
the change to Gerrit for review.
|
||||
Zuul's configuration is stored in :config:`zuul/main.yaml`. Anyone
|
||||
may propose a change to the configuration by editing that file and
|
||||
submitting the change to Gerrit for review.
|
||||
|
||||
For the full syntax of Zuul's configuration file format, see the `Zuul
|
||||
reference manual`_.
|
||||
@ -63,33 +95,52 @@ reference manual`_.
|
||||
Sysadmin
|
||||
========
|
||||
|
||||
Zuul and gear are lightweight - it should be possible to run both on a
|
||||
1G instance for small deployments. OpenStack's deployment requires at
|
||||
least a 8G instance at the time of writing, though additional cache
|
||||
memory helps performance.
|
||||
Zuul has three main subsystems:
|
||||
|
||||
Zuul is mostly stateless, so the server does not need backing up (though
|
||||
it does rely on a Trove instance for its build history). However zuul
|
||||
talks through git and ssh so you will need to manually check ssh host
|
||||
keys as the zuul user. e.g.::
|
||||
* Zuul Scheduler
|
||||
* Zuul Executors
|
||||
* Zuul Web
|
||||
|
||||
that in OpenStack's deployment depend on four 'external' systems:
|
||||
|
||||
* Nodepool
|
||||
* Zookeeper
|
||||
* gear
|
||||
* MySQL
|
||||
|
||||
Scheduler
|
||||
---------
|
||||
|
||||
The Zuul Scheduler and gear are all co-located on a single host,
|
||||
referred to by the ``zuul.openstack.org`` CNAME in DNS.
|
||||
|
||||
Zuul is stateless, so the server does not need backing up. However
|
||||
zuul talks through git and ssh so you will need to manually check ssh
|
||||
host keys as the zuul user.
|
||||
|
||||
.. note:: Is this still true or are we managing host keys in puppet now?
|
||||
|
||||
e.g.::
|
||||
|
||||
sudo su - zuul
|
||||
ssh -p 29418 review.opendev.org
|
||||
|
||||
To debug Zuul's gearman server, SSL is required. Use the following
|
||||
command::
|
||||
The Zuul Scheduler talks to Nodepool using Zookeeper and distributes work to
|
||||
the executors using gear.
|
||||
|
||||
openssl s_client -connect localhost:4730 -cert /etc/zuul/ssl/client.pem -key /etc/zuul/ssl/client.key
|
||||
OpenStack's Zuul installation is also configured to write job results into
|
||||
a MySQL database via the SQL Reporter plugin. The database for that is a
|
||||
Rackspace Cloud DB and is configured in the ``mysql`` entry of the
|
||||
``zuul_connection_secrets`` entry for the ``zuul-scheduler`` group.
|
||||
|
||||
Restarts
|
||||
--------
|
||||
Restarting the Scheduler
|
||||
------------------------
|
||||
|
||||
Zuul restarts are disruptive, so non-emergency restarts should always be
|
||||
scheduled for quieter times of the day, week and cycle. To be as
|
||||
courteous to developers as possible, just prior to a restart the `OpenStack
|
||||
Status Page <https://zuul.opendev.org/t/openstack/status>`_ should be checked to
|
||||
see the status of the OpenStack gate. If there is a series of changes nearly
|
||||
merged, wait until that has been completed.
|
||||
Zuul Scheduler restarts are disruptive, so non-emergency restarts should
|
||||
always be scheduled for quieter times of the day, week and cycle. To be as
|
||||
courteous to developers as possible, just prior to a restart the `Zuul
|
||||
Status Page`_ should be checked to see the status of the gate. If there is a
|
||||
series of changes nearly merged, wait until that has been completed.
|
||||
|
||||
Since Zuul is stateless, some work needs to be done to save and then
|
||||
re-enqueue patches when restarts are done. To accomplish this, start by
|
||||
@ -105,51 +156,72 @@ to save the check and gate queues::
|
||||
These check.sh and gate.sh scripts will be used after the restart to
|
||||
re-enqueue the changes.
|
||||
|
||||
Now use `service zuul-scheduler stop` to stop zuul and then run ps to
|
||||
make sure the process has actually stopped, it may take several seconds
|
||||
for it to finally go away.
|
||||
Now use `service zuul stop` to stop zuul and then run ps to make sure
|
||||
the process has actually stopped, it may take several seconds for it to
|
||||
finally go away.
|
||||
|
||||
When you are satisfied that zuul is up, first run the gate.sh script and
|
||||
then check.sh to re-enqueue the changes from before the restart::
|
||||
Once you're ready, use `service zuul start` to start zuul again.
|
||||
|
||||
To re-enqueue saved jobs, first run the gate.sh script and then check.sh to
|
||||
re-enqueue the changes from before the restart::
|
||||
|
||||
./gate.sh
|
||||
./check.sh
|
||||
|
||||
You may watch the `Zuul Status Page
|
||||
<https://zuul.opendev.org/t/openstack/status>`_ to confirm that changes are
|
||||
returning to the queues. This frontend is provided by the zuul-web
|
||||
service on the same server, which may also need to be restarted.
|
||||
You may watch the `Zuul Status Page`_ to confirm that changes are
|
||||
returning to the queues.
|
||||
|
||||
Executors
|
||||
---------
|
||||
|
||||
Servers with names matching the pattern ze*.openstack.org are Zuul
|
||||
Executors. These are horizontally scalable components of Zuul which
|
||||
run Ansible within a Bubblewrap context and connect to job nodes.
|
||||
They can be started and stopped at will, and new ones added as
|
||||
necessary to accommodate load.
|
||||
The Zuul Executors are a horizontally scalable set of servers named
|
||||
ze*.openstack.org. They perform git merging operations for the scheduler
|
||||
and execute Ansible playbooks to actually run jobs.
|
||||
|
||||
Mergers
|
||||
-------
|
||||
Our jobs are configured to upload as much information as possible along with
|
||||
their logs, but if there is an error which can not be diagnosed in that
|
||||
manner, logs are available in the executor-debug log file on
|
||||
the executor host. You may use the Zuul build UUID to track
|
||||
assignment of a given job from the Zuul scheduler to the Zuul executor
|
||||
used by that job.
|
||||
|
||||
Servers with names matching the pattern zm*.openstack.org are Zuul
|
||||
Mergers. These are horizontally scalable components of Zuul which
|
||||
perform git operations for the benefit of jobs. They can be started
|
||||
and stopped at will, and new ones added as necessary to accommodate
|
||||
load.
|
||||
It is safe, although not free, to restart executors. If an executor goes away
|
||||
the scheduler will reschedule the jobs it was originally running.
|
||||
|
||||
Secrets
|
||||
-------
|
||||
Web
|
||||
---
|
||||
|
||||
In some cases it may be warranted to compare the decrypted plaintext of
|
||||
a secret from job configuration against a reference value while
|
||||
troubleshooting, since random padding means encrypting the same
|
||||
plaintext a second time will result in wholly different ciphertext. In
|
||||
order to avoid unintentional disclosure this should only be done when
|
||||
absolutely necessary, but it's possible to decrypt a secret locally on
|
||||
the scheduler server with a command like the following (just extract the
|
||||
secret ciphertext from the job configuration first to remove surrounding
|
||||
YAML, there is no need to dedent nor recombine split lines)::
|
||||
Zuul Web is a horizontally scalable service. It is currently running colocated
|
||||
with the scheduler on zuul.openstack.org. Zuul Web provides live console
|
||||
streaming and will be the home of various web dashboards such as the status
|
||||
page.
|
||||
|
||||
cat ciphertext.txt | base64 -d | sudo openssl rsautl -decrypt -oaep -inkey \
|
||||
/var/lib/zuul/keys/secrets/project/gerrit/openstack/project-config/0.pem
|
||||
Zuul Web is stateless so is safe to restart, however restarting it will result
|
||||
in a loss of connection for anyone watching a live-stream of a console log
|
||||
when the restart happens.
|
||||
|
||||
.. _zuul_github_projects:
|
||||
|
||||
GitHub Projects
|
||||
===============
|
||||
|
||||
OpenStack does not use GitHub for development purposes, but there are some
|
||||
non-OpenStack projects in the broader ecosystem that we care about who do.
|
||||
When we are interested in setting up jobs in Zuul to test the interaction
|
||||
between OpenStack projects and those ecosystem projects, we can add the
|
||||
OpenDev Zuul GitHub app to those projects, then configure them in Zuul.
|
||||
|
||||
In order to add the GitHub app to a project, an admin on that project should
|
||||
navigate to the `OpenDev Zuul`_ app in the GitHub UI. From there they can
|
||||
click "Install", then choose the project or organization they want to install
|
||||
the App on.
|
||||
|
||||
The repository then needs to be added to the `zuul/main.yaml` file before Zuul
|
||||
can be configured to actually run jobs on it.
|
||||
|
||||
Information about the configuration of the OpenDev Zuul App itself can be
|
||||
found on the :ref:`github` page at :ref:`openstack_zuul_app`.
|
||||
|
||||
.. _OpenDev Zuul: https://github.com/apps/opendev-zuul
|
||||
.. _Zuul Reference Manual: https://zuul-ci.org/docs/zuul
|
||||
.. _Zuul Status Page: http://zuul.opendev.org
|
||||
|
||||
@ -1,228 +0,0 @@
|
||||
:title: Zuulv3
|
||||
|
||||
.. _zuulv3:
|
||||
|
||||
Zuul v3
|
||||
#######
|
||||
|
||||
.. note:: Zuul v3 is the upcoming release of Zuul. While it is not in
|
||||
production yet, we are running it. Once it goes live, this
|
||||
document should be renamed to Zuul and the old document should
|
||||
go away - so don't develop any emotional attachments to permalinks
|
||||
to this document.
|
||||
|
||||
Zuul is a pipeline-oriented project gating system. It facilitates
|
||||
running tests and automated tasks in response to Code Review events.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* http://zuul.openstack.org
|
||||
* zuul*.openstack.org
|
||||
* ze*.openstack.org
|
||||
* zm*.openstack.org
|
||||
:Puppet:
|
||||
* https://opendev.org/opendev/puppet-zuul
|
||||
* https://opendev.org/opendev/puppet-openstackci/src/branch/master/manifests/zuul_merger.pp
|
||||
* https://opendev.org/opendev/puppet-openstackci/src/branch/master/manifests/zuul_scheduler.pp
|
||||
:Configuration:
|
||||
* :config:`zuul/main.yaml`
|
||||
* :config:`zuul.d`
|
||||
:Projects:
|
||||
* https://opendev.org/zuul/zuul
|
||||
:Bugs:
|
||||
* https://storyboard.openstack.org/#!/project/679
|
||||
:Resources:
|
||||
* `Zuul Reference Manual`_
|
||||
:Chat:
|
||||
* #zuul on freenode
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
The OpenStack project uses a number of pipelines in Zuul:
|
||||
|
||||
**check**
|
||||
Newly uploaded patchsets enter this pipeline to receive an initial
|
||||
+/-1 Verified vote.
|
||||
|
||||
**gate**
|
||||
Changes that have been approved by core reviewers are enqueued in
|
||||
order in this pipeline, and if they pass tests, will be merged.
|
||||
|
||||
**post**
|
||||
This pipeline runs jobs that operate after each change is merged.
|
||||
|
||||
**pre-release**
|
||||
This pipeline runs jobs on projects in response to pre-release tags.
|
||||
|
||||
**release**
|
||||
When a commit is tagged as a release, this pipeline runs jobs that
|
||||
publish archives and documentation.
|
||||
|
||||
**silent**
|
||||
This pipeline is used for silently testing new jobs.
|
||||
|
||||
**experimental**
|
||||
This pipeline is used for on-demand testing of new jobs.
|
||||
|
||||
**periodic**
|
||||
This pipeline has jobs triggered on a timer for e.g. testing for
|
||||
environmental changes daily.
|
||||
|
||||
Zuul watches events in Gerrit (using the Gerrit "stream-events"
|
||||
command) and matches those events to the pipelines above. If a match
|
||||
is found, it adds the change to the pipeline and starts running
|
||||
related jobs.
|
||||
|
||||
The **gate** pipeline uses speculative execution to improve
|
||||
throughput. Changes are tested in parallel under the assumption that
|
||||
changes ahead in the queue will merge. If they do not, Zuul will
|
||||
abort and restart tests without the affected changes. This means that
|
||||
many changes may be tested in parallel while continuing to assure that
|
||||
each commit is correctly tested.
|
||||
|
||||
Zuul's current status may be viewed at
|
||||
`<http://zuul.openstack.org/>`_.
|
||||
|
||||
Zuul's configuration is stored in :config:`zuul/main.yaml`. Anyone
|
||||
may propose a change to the configuration by editing that file and
|
||||
submitting the change to Gerrit for review.
|
||||
|
||||
For the full syntax of Zuul's configuration file format, see the `Zuul
|
||||
reference manual`_.
|
||||
|
||||
Sysadmin
|
||||
========
|
||||
|
||||
Zuul has three main subsystems:
|
||||
|
||||
* Zuul Scheduler
|
||||
* Zuul Executors
|
||||
* Zuul Web
|
||||
|
||||
that in OpenStack's deployment depend on four 'external' systems:
|
||||
|
||||
* Nodepool
|
||||
* Zookeeper
|
||||
* gear
|
||||
* MySQL
|
||||
|
||||
Scheduler
|
||||
---------
|
||||
|
||||
The Zuul Scheduler and gear are all co-located on a single host,
|
||||
referred to by the ``zuul.openstack.org`` CNAME in DNS.
|
||||
|
||||
Zuul is stateless, so the server does not need backing up. However
|
||||
zuul talks through git and ssh so you will need to manually check ssh
|
||||
host keys as the zuul user.
|
||||
|
||||
.. note:: Is this still true or are we managing host keys in puppet now?
|
||||
|
||||
e.g.::
|
||||
|
||||
sudo su - zuul
|
||||
ssh -p 29418 review.opendev.org
|
||||
|
||||
The Zuul Scheduler talks to Nodepool using Zookeeper and distributes work to
|
||||
the executors using gear.
|
||||
|
||||
OpenStack's Zuul installation is also configured to write job results into
|
||||
a MySQL database via the SQL Reporter plugin. The database for that is a
|
||||
Rackspace Cloud DB and is configured in the ``mysql`` entry of the
|
||||
``zuul_connection_secrets`` entry for the ``zuul-scheduler`` group.
|
||||
|
||||
Restarting the Scheduler
|
||||
------------------------
|
||||
|
||||
Zuul Scheduler restarts are disruptive, so non-emergency restarts should
|
||||
always be scheduled for quieter times of the day, week and cycle. To be as
|
||||
courteous to developers as possible, just prior to a restart the `Zuul
|
||||
Status Page`_ should be checked to see the status of the gate. If there is a
|
||||
series of changes nearly merged, wait until that has been completed.
|
||||
|
||||
Since Zuul is stateless, some work needs to be done to save and then
|
||||
re-enqueue patches when restarts are done. To accomplish this, start by
|
||||
running `zuul-changes.py
|
||||
<https://opendev.org/zuul/zuul/src/branch/master/tools/zuul-changes.py>`_
|
||||
to save the check and gate queues::
|
||||
|
||||
python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
|
||||
check >check.sh
|
||||
python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
|
||||
gate >gate.sh
|
||||
|
||||
These check.sh and gate.sh scripts will be used after the restart to
|
||||
re-enqueue the changes.
|
||||
|
||||
Now use `service zuul stop` to stop zuul and then run ps to make sure
|
||||
the process has actually stopped, it may take several seconds for it to
|
||||
finally go away.
|
||||
|
||||
Once you're ready, use `service zuul start` to start zuul again.
|
||||
|
||||
To re-enqueue saved jobs, first run the gate.sh script and then check.sh to
|
||||
re-enqueue the changes from before the restart::
|
||||
|
||||
./gate.sh
|
||||
./check.sh
|
||||
|
||||
You may watch the `Zuul Status Page`_ to confirm that changes are
|
||||
returning to the queues.
|
||||
|
||||
Executors
|
||||
---------
|
||||
|
||||
The Zuul Executors are a horizontally scalable set of servers named
|
||||
ze*.openstack.org. They perform git merging operations for the scheduler
|
||||
and execute Ansible playbooks to actually run jobs.
|
||||
|
||||
Our jobs are configured to upload as much information as possible along with
|
||||
their logs, but if there is an error which can not be diagnosed in that
|
||||
manner, logs are available in the executor-debug log file on
|
||||
the executor host. You may use the Zuul build UUID to track
|
||||
assignment of a given job from the Zuul scheduler to the Zuul executor
|
||||
used by that job.
|
||||
|
||||
It is safe, although not free, to restart executors. If an executor goes away
|
||||
the scheduler will reschedule the jobs it was originally running.
|
||||
|
||||
Web
|
||||
---
|
||||
|
||||
Zuul Web is a horizontally scalable service. It is currently running colocated
|
||||
with the scheduler on zuul.openstack.org. Zuul Web provides live console
|
||||
streaming and will be the home of various web dashboards such as the status
|
||||
page.
|
||||
|
||||
Zuul Web is stateless so is safe to restart, however restarting it will result
|
||||
in a loss of connection for anyone watching a live-stream of a console log
|
||||
when the restart happens.
|
||||
|
||||
.. _zuul_github_projects:
|
||||
|
||||
GitHub Projects
|
||||
===============
|
||||
|
||||
OpenStack does not use GitHub for development purposes, but there are some
|
||||
non-OpenStack projects in the broader ecosystem that we care about who do.
|
||||
When we are interested in setting up jobs in Zuul to test the interaction
|
||||
between OpenStack projects and those ecosystem projects, we can add the
|
||||
OpenDev Zuul GitHub app to those projects, then configure them in Zuul.
|
||||
|
||||
In order to add the GitHub app to a project, an admin on that project should
|
||||
navigate to the `OpenDev Zuul`_ app in the GitHub UI. From there they can
|
||||
click "Install", then choose the project or organization they want to install
|
||||
the App on.
|
||||
|
||||
The repository then needs to be added to the `zuul/main.yaml` file before Zuul
|
||||
can be configured to actually run jobs on it.
|
||||
|
||||
Information about the configuration of the OpenDev Zuul App itself can be
|
||||
found on the :ref:`github` page at :ref:`openstack_zuul_app`.
|
||||
|
||||
.. _OpenDev Zuul: https://github.com/apps/opendev-zuul
|
||||
.. _Zuul Reference Manual: https://docs.openstack.org/infra/zuul
|
||||
.. _Zuul Status Page: http://zuul.openstack.org
|
||||
Loading…
x
Reference in New Issue
Block a user