Update Zuul service documentation

Much of this document was stale since putting Zuul v3 into
production. Attempt to bring it up to our current state.

Change-Id: I1b9d665cbf5ca88917d6e0361a6b16026b2ad6c6

doc/source/zuul.rst

@@ -13,54 +13,29 @@ At a Glance
 
 :Hosts:
   * https://zuul.openstack.org
+  * ze*.openstack.org
   * zm*.openstack.org
 :Puppet:
   * https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/
   * :cgit_file:`modules/openstack_project/manifests/zuul_prod.pp`
   * :cgit_file:`modules/openstack_project/manifests/zuul_dev.pp`
 :Configuration:
-  * :config:`zuul/layout.yaml`
+  * :config:`zuul/`
+  * :config:`zuul.d/`
 :Projects:
-  * https://git.openstack.org/cgit/openstack-infra/zuul
+  * https://git.zuul-ci.org/cgit/zuul
 :Bugs:
-  * https://storyboard.openstack.org/#!/project/679
+  * https://storyboard.openstack.org/#!/project/openstack-infra/zuul
 :Resources:
-  * `Zuul Reference Manual <http://docs.openstack.org/infra/zuul>`_
+  * `Zuul Reference Manual <https://zuul-ci.org/docs/zuul>`_
 :Chat:
   * #zuul on freenode
 
 Overview
 ========
 
-The OpenStack project uses a number of pipelines in Zuul:
+The OpenStack project uses a number of pipelines in Zuul, as defined
-
+in :config:`zuul.d/pipelines.yaml`.
-**check**
-  Newly uploaded patchsets enter this pipeline to receive an initial
-  +/-1 Verified vote.
-
-**gate**
-  Changes that have been approved by core reviewers are enqueued in
-  order in this pipeline, and if they pass tests, will be merged.
-
-**post**
-  This pipeline runs jobs that operate after each change is merged.
-
-**pre-release**
-  This pipeline runs jobs on projects in response to pre-release tags.
-
-**release**
-  When a commit is tagged as a release, this pipeline runs jobs that
-  publish archives and documentation.
-
-**silent**
-  This pipeline is used for silently testing new jobs.
-
-**experimental**
-  This pipeline is used for on-demand testing of new jobs.
-
-**periodic**
-  This pipeline has jobs triggered on a timer for e.g. testing for
-  environmental changes daily.
 
 Zuul watches events in Gerrit (using the Gerrit "stream-events"
 command) and matches those events to the pipelines above.  If a match
@@ -77,9 +52,10 @@ each commit is correctly tested.
 Zuul's current status may be viewed at
 `<https://zuul.openstack.org/>`_.
 
-Zuul's configuration is stored in :config:`zuul/layout.yaml`.  Anyone
+Zuul's configuration is distributed across projects listed in
-may propose a change to the configuration by editing that file and
+:config:`zuul/main.yaml`.  Anyone may propose a change to the
-submitting the change to Gerrit for review.
+configuration by editing configuration in those projects and submitting
+the change to Gerrit for review.
 
 For the full syntax of Zuul's configuration file format, see the `Zuul
 reference manual`_.
@@ -89,11 +65,13 @@ Sysadmin
 
 Zuul and gear are lightweight - it should be possible to run both on a
 1G instance for small deployments. OpenStack's deployment requires at
-least a 2G instance at the time of writing.
+least a 8G instance at the time of writing, though additional cache
+memory helps performance.
 
-Zuul is stateless, so the server does not need backing up. However
+Zuul is mostly stateless, so the server does not need backing up (though
-zuul talks through git and ssh so you will need to manually check ssh
+it does rely on a Trove instance for its build history). However zuul
-host keys as the zuul user. e.g.::
+talks through git and ssh so you will need to manually check ssh host
+keys as the zuul user. e.g.::
 
   sudo su - zuul
   ssh -p 29418 review.openstack.org
@@ -116,7 +94,7 @@ merged, wait until that has been completed.
 Since Zuul is stateless, some work needs to be done to save and then
 re-enqueue patches when restarts are done. To accomplish this, start by
 running `zuul-changes.py
-<https://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/zuul-changes.py>`_
+<https://git.zuul-ci.org/cgit/zuul/tree/tools/zuul-changes.py>`_
 to save the check and gate queues::
 
   python /opt/zuul/tools/zuul-changes.py http://zuul.openstack.org \
@@ -127,40 +105,35 @@ to save the check and gate queues::
 These check.sh and gate.sh scripts will be used after the restart to
 re-enqueue the changes.
 
-Now use `service zuul stop` to stop zuul and then run ps to make sure
+Now use `service zuul-scheduler stop` to stop zuul and then run ps to
-the process has actually stopped, it may take several seconds for it to
+make sure the process has actually stopped, it may take several seconds
-finally go away.
+for it to finally go away.
 
-With Zuul stopped, delete all the used nodes in nodepool. Wait for one
+When you are satisfied that zuul is up, first run the gate.sh script and
-of each variety to come up before using `service zuul start` to start
+then check.sh to re-enqueue the changes from before the restart::
-zuul again.
-
-Once Zuul is started, run netcat against localhost 4730 port to confirm
-that all the node types (particularly the uncommon ones) are registered
-with Gearman before re-enqueuing patches. For instance::
-
-  echo "status" | nc localhost 4730 | grep :centos7
-
-When you are satisfied that all the node types have returned, first run
-the gate.sh script and then check.sh to re-enqueue the changes from
-before the restart::
 
   ./gate.sh
   ./check.sh
 
 You may watch the `Zuul Status Page
 <https://zuul.openstack.org/>`_ to confirm that changes are
-returning to the queues.
+returning to the queues. This frontend is provided by the zuul-web
+service on the same server, which may also need to be restarted.
+
+Executors
+---------
+
+Servers with names matching the pattern ze*.openstack.org are Zuul
+Executors.  These are horizontally scalable components of Zuul which
+run Ansible within a Bubblewrap context and connect to job nodes.
+They can be started and stopped at will, and new ones added as
+necessary to accommodate load.
 
 Mergers
 -------
 
 Servers with names matching the pattern zm*.openstack.org are Zuul
 Mergers.  These are horizontally scalable components of Zuul which
-perform git operations for the benefit of jobs.  They serve git
+perform git operations for the benefit of jobs. They can be started
-repositories via Apache over http, and jobs fetch changes to test from
+and stopped at will, and new ones added as necessary to accommodate
-them.  They can be started and stopped at will, and new ones added as
+load.
-necessary to accommodate load.  If you remove a merger, be sure to
-leave Apache running for several hours until the last job that may
-have been launched with instructions to fetch from that merger has
-completed.