docs: update console streaming docs

Update the developer Ansible docs to give some more details on how the
zuul_console daemon streaming happens.  In a couple of places where it
is mentioned, rework things and point them at this explaination.

Change-Id: I5bfb61323bf3219168d4d014cbb9703eed230e71

doc/source/developer/ansible.rst

@@ -4,19 +4,11 @@ Ansible Integration
 Zuul contains Ansible modules and plugins to control the execution of Ansible
 Job content.
 
-Build Log Support
------------------
+Zuul provides realtime build log streaming to end users so that users
+can watch long-running jobs in progress.
 
-Zuul provides realtime build log streaming to end users so that users can
-watch long-running jobs in progress. As jobs may be written that execute a
-shell script that could run for a long time, additional effort is expended
-to stream stdout and stderr of shell tasks as they happen rather than waiting
-for the command to finish.
-
-Zuul contains a modified version of the :ansible:module:`command`
-that starts a log streaming daemon on the build node.
-
-.. automodule:: zuul.ansible.base.library.command
+Streaming job output
+--------------------
 
 All jobs run with the :py:mod:`zuul.ansible.base.callback.zuul_stream` callback
 plugin enabled, which writes the build log to a file so that the
@@ -35,10 +27,55 @@ exposes that log stream over a websocket connection as part of
 In addition to real-time streaming, Zuul also installs another callback module,
 :py:mod:`zuul.ansible.base.callback.zuul_json.CallbackModule` that collects all
 of the information about a given run into a json file which is written to the
-work dir so that it can be published along with build logs. Since the streaming
-log is by necessity a single text stream, choices have to be made for
-readability about what data is shown and what is not shown. The json log file
-is intended to allow for a richer more interactive set of data to be displayed
-to the user.
+work dir so that it can be published along with build logs.
 
 .. autoclass:: zuul.ansible.base.callback.zuul_json.CallbackModule
+
+Since the streaming log is by necessity a single text stream, choices
+have to be made for readability about what data is shown and what is
+not shown. The json log file is intended to allow for a richer more
+interactive set of data to be displayed to the user.
+
+.. _zuul_console_streaming:
+
+Capturing live command output
+-----------------------------
+
+As jobs may execute long-running shell scripts or other commands,
+additional effort is expended to stream ``stdout`` and ``stderr`` of
+shell tasks as they happen rather than waiting for the command to
+finish.
+
+The global job configuration should run the ``zuul_console`` task as a
+very early prerequisite step.
+
+.. automodule:: zuul.ansible.base.library.zuul_console
+
+This will start a daemon that listens on TCP port 19885 on the testing
+node.  This daemon can be queried to stream back the output of shell
+tasks as described below.
+
+Zuul contains a modified version of Ansible's
+:ansible:module:`command` module that overrides the default
+implementation.
+
+.. automodule:: zuul.ansible.base.library.command
+
+This library will capture the output of the running
+command and write it to a temporary file on the host the command is
+running on.  These files are named in the format
+``/tmp/console-<uuid>-<task_id>-<host>.log``
+
+The ``zuul_stream`` callback mentioned above will send a request to
+the remote ``zuul_console`` daemon, providing the uuid and task id of
+the task it is currently processing.  The ``zuul_console`` daemon will
+then read the logfile from disk and stream the data back as it
+appears, which ``zuul_stream`` will then present as described above.
+
+The ``zuul_stream`` callback will indicate to the ``zuul_console``
+daemon when it has finished reading the task, which prompts the remote
+side to remove the temporary streaming output files.  In some cases,
+aborting the Ansible process may not give the ``zuul_stream`` callback
+the chance to send this notice, leaking the temporary files.  If nodes
+are ephemeral this makes little difference, but these files may be
+visible on static nodes.

doc/source/howtos/nodepool_static.rst

@@ -15,9 +15,9 @@ the following requirements:
 
 * Must be reachable by Zuul executors and have SSH access enabled.
 * Must have a user that Zuul can use for SSH.
-* Must have Python 2 installed for Ansible.
-* Must be reachable by Zuul executors over TCP port 19885 (console log
-  streaming).
+* Must have an Ansible supported Python installed
+* Must be reachable by Zuul executors over TCP port 19885 for console
+  log streaming.  See :ref:`nodepool_console_streaming`
 
 When setting up your nodepool.yaml file, you will need the host keys
 for each node for the ``host-key`` value. This can be obtained with
@@ -40,7 +40,7 @@ nodes.  Place this file in ``/etc/nodepool/nodepool.yaml``:
      - host: localhost
 
    labels:
-     - name: ubuntu-xenial
+     - name: ubuntu-jammy
 
    providers:
      - name: static-vms
@@ -49,14 +49,34 @@ nodes.  Place this file in ``/etc/nodepool/nodepool.yaml``:
          - name: main
            nodes:
              - name: 192.168.1.10
-               labels: ubuntu-xenial
+               labels: ubuntu-jammy
                host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ4qE47o5gb"
                username: zuul
              - name: 192.168.1.11
-               labels: ubuntu-xenial
+               labels: ubuntu-jammy
                host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ5sE47o5gc"
                username: zuul
    EOF"
 
 Make sure that ``username``, ``host-key``, IP addresses and label names are
 customized for your environment.
+
+.. _nodepool_console_streaming:
+
+Log streaming
+-------------
+
+The log streaming service enables Zuul to show the live status of
+long-running ``shell`` or ``command`` tasks.  The server side is setup
+by the ``zuul_console:`` task built-in to Zuul's Ansible installation.
+The executor requires the ability to communicate with the job nodes on
+port 19885 for this to work.
+
+The log streaming service may leave files on the static node in the
+format ``/tmp/console-<uuid>-<task_id>-<host>.log`` if jobs are
+interrupted.  These may be safely removed after a short period of
+inactivity with a command such as
+
+.. code-block:: shell
+
+   find /tmp -maxdepth 1 -name 'console-*-*-<host>.log' -mtime +2 -delete

doc/source/installation.rst

@@ -10,11 +10,15 @@ Nodepool
 ~~~~~~~~
 
 In order to run all but the simplest jobs, Zuul uses a companion
-program, Nodepool, to supply the nodes (whether dynamic cloud
-instances or static hardware) used by jobs.  Before starting Zuul,
-ensure you have Nodepool installed and any images you require built.
-Zuul only makes one requirement of these nodes: that it be able to log
-in given a username and ssh private key.
+program `Nodepool <https://opendev.org/zuul/nodepool>`__ to supply the
+nodes (whether dynamic cloud instances or static hardware) used by
+jobs.  Before starting Zuul, ensure you have Nodepool installed and
+any images you require built.
+
+Zuul must be able to log into the nodes provisioned by Nodepool with a
+given username and SSH private key.  Executors should also be able to
+talk to nodes on TCP port 19885 for log streaming; see
+:ref:`nodepool_console_streaming`.
 
 ZooKeeper
 ~~~~~~~~~