Document the OpenDev docker image jobs

Change-Id: I4d3a7156fb3275049e58c535a2a1beee13e17766

doc/source/base.rst

@@ -0,0 +1,6 @@
+Base Jobs
+=========
+
+.. zuul:autojob:: base
+
+.. zuul:autojob:: base-test

doc/source/conf.py

@@ -19,6 +19,8 @@
 extensions = [
     # 'sphinx.ext.autodoc',
     # 'sphinx.ext.intersphinx',
+    'sphinxcontrib.blockdiag',
+    'sphinxcontrib.seqdiag',
     'zuul_sphinx',
 ]
 

doc/source/docker-image.rst

@@ -0,0 +1,165 @@
+Container Images
+================
+
+The jobs described in this section all work together to handle the
+full gating process for continuously deployed container images.  They
+can be used to build or test images which rely on other images using
+the full power of Zuul's speculative execution.
+
+There are a few key concepts to keep in mind:
+
+A *buildset* is a group of jobs all running on the same change.
+
+A *buildset registry* is a container image registry which is used to
+store speculatively built images for the use of jobs in a single
+buildset.  It holds the differences between the current state of the
+world and the future state if the change in question (and all of its
+dependent changes) were to merge.  It must be started by one of the
+jobs in a buildset, and it ceases to exist once that job is complete.
+
+An *intermediate registry* is a long-running registry that is used to
+store images created for unmerged changes for use by other unmerged
+changes.  It is not publicly accessible and is intended only to be
+used by Zuul in order to transfer artifacts from one buildset to
+another.  OpenDev maintains such a registry.
+
+With these concepts in mind, the jobs described below implement the
+following workflow for a single change:
+
+.. _buildset_image_transfer:
+
+.. seqdiag::
+   :caption: Buildset registry image transfer
+
+   seqdiag image_transfer {
+     Ireg [label="Intermediate\nRegistry"];
+     Breg [label="Buildset\nRegistry"];
+     Bjob [label="Image Build Job"];
+     Djob [label="Deployment Test Job"];
+
+     Ireg -> Breg [label='Images from previous changes'];
+     Breg -> Bjob [label='Images from previous changes'];
+     Breg <- Bjob [label='Current image'];
+     Ireg <- Breg [noactivate, label='Current image'];
+     Breg -> Djob [label='Current and previous images'];
+     Breg <- Djob [style=none];
+     Ireg <- Breg [style=none];
+   }
+
+The intermediate registry is always running and the buildset registry
+is started by a job running on a change.  The "Image Build" and
+"Deployment Test" jobs are example jobs which might be running on a
+change.  Essentially, these are image producer or consumer jobs
+respectively.
+
+There are two ways to use the jobs described below:
+
+A Repository with Producers and Consumers
+-----------------------------------------
+
+The first is in a repository where images are both produced and
+consumed.  In this case, we can expect that there will be at least one
+image build job, and at least one job which uses that image (for
+example, by performing a test deployment of the image).  In this case
+we need to construct a job graph with dependencies as follows:
+
+.. blockdiag::
+
+   blockdiag dependencies {
+     obr [label='opendev-\nbuildset-registry'];
+     bi [label='build-image'];
+     ti [label='test-image'];
+
+     obr <- bi <- ti;
+   }
+
+The :zuul:job:`opendev-buildset-registry` job will run first and
+automatically start a buildset registry populated with images built
+from any changes which appear ahead of the current change.  It will
+then return its connection information to Zuul and pause and continue
+running until the completion of the build and test jobs.
+
+The build-image job should inherit from
+:zuul:job:`opendev-build-docker-image`, which will ensure that it is
+automatically configured to use the buildset registry.
+
+The test-image job is something that you will create yourself.  There
+is no standard way to test or deploy an image, that depends on your
+application.  However, there is one thing you will need to do in your
+job to take advantage of the buildset registry.  In a pre-run playbook,
+use the `use-buildset-registry
+<https://zuul-ci.org/docs/zuul-jobs/roles.html#role-use-buildset-registry>`_
+role:
+
+.. code:: yaml
+
+   - hosts: all
+     roles:
+       - use-buildset-registry
+
+That will configure the docker daemon on the host to use the buildset
+registry so that it will use the newly built version of any required
+images.
+
+A Repository with Only Producers
+--------------------------------
+
+The second way to use these jobs is in a repository where an image is
+merely built, but not deployed.  In this case, there are no consumers
+of the buildset registry other than the image build job, and so the
+registry can be run on the job itself.  In this case, you may omit the
+:zuul:job:`opendev-buildset-registry` job and run only the
+:zuul:job:`opendev-build-docker-image` job.
+
+Publishing an Image
+-------------------
+
+So far we've covered the image building process.  This system also
+provides two more jobs that are used in publishing images to Docker
+Hub.
+
+The :zuul:job:`opendev-upload-docker-image` job does everything the
+:zuul:job:`opendev-build-docker-image` job does, but it also uploads
+the built image to Docker Hub using an automatically-generated and
+temporary tag.  The "build" job is designed to be used in the
+*check* pipeline, while the "upload" job is designed to take its
+place in the *gate* pipeline.  By front-loading the upload to Docker
+Hub, we reduce the chance that a credential or network error will
+prevent us from publishing an image after a change lands.
+
+The :zuul:job:`opendev-promote-docker-image` jobs is designed to be
+used in the *promote* pipeline and simply re-tags the image on Docker
+Hub after the change lands.
+
+Keeping in mind that everything described above in
+:ref:`buildset_image_transfer` applies to the
+:zuul:job:`opendev-upload-docker-image` job, the following illustrates
+the additional tasks performed by the "upload" and "promote" jobs:
+
+.. seqdiag::
+
+   seqdiag image_transfer {
+     DH [activated, label="Docker Hub"];
+     Ujob [label="upload-image"];
+     Pjob [label="promote-image"];
+
+     DH -> Ujob [style=none];
+     DH <- Ujob [label='Current image with temporary tag'];
+     DH -> Pjob [label='Current image manifest with temporary tag',
+                 note='Only the manifest
+                       is transferred,
+                       not the actual
+                       image layers.'];
+     DH <- Pjob [label='Current image manifest with final tag'];
+   }
+
+Jobs
+----
+
+.. zuul:autojob:: opendev-buildset-registry
+
+.. zuul:autojob:: opendev-build-docker-image
+
+.. zuul:autojob:: opendev-upload-docker-image
+
+.. zuul:autojob:: opendev-promote-docker-image

doc/source/index.rst

@@ -1,7 +1,7 @@
 .. include:: ../../README.rst
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
    jobs
    roles

doc/source/jobs.rst

@@ -1,5 +1,10 @@
 Jobs
-=====
+====
 
-.. zuul:autojobs::
+Documentation for jobs is grouped by subject area:
 
+.. toctree::
+   :maxdepth: 1
+
+   base
+   docker-image

test-requirements.txt

@@ -17,3 +17,5 @@ ansible-lint<4
 bashate>=0.2
 zuul-sphinx>=0.1.1
 stestr>=1.0.0 # Apache-2.0
+sphinxcontrib-blockdiag>=1.1.0
+sphinxcontrib-seqdiag

zuul.yaml

@@ -164,9 +164,16 @@
 - job:
     name: opendev-buildset-registry
     description: |
-      Inherit from this job to get a buildset registry which interacts
-      with the intermediate CI registry to share speculative container
-      images between projects.
+      Starts a buildset registry which interacts with the intermediate
+      CI registry to share speculative container images between
+      projects.
+
+      Configure any jobs which require the use of a buildset registry
+      to depend on this job using the "dependencies" job attribute.
+
+      This job will pause after starting the registry so that it is
+      available to any jobs which depend on it.  Once all such jobs
+      are complete, this job will finish.
     pre-run: playbooks/buildset-registry/pre.yaml
     run: playbooks/buildset-registry/run.yaml
     post-run: playbooks/buildset-registry/post.yaml
@@ -179,7 +186,10 @@
     name: opendev-build-docker-image
     parent: opendev-buildset-registry
     description: |
-      Starts a buildset registry and builds one or more docker images.
+      Starts a buildset registry (if one has not already been started,
+      e.g., by invoking :zuul:job:`opendev-buildset-registry` and
+      specifying it as a dependency) and builds one or more docker
+      images.
 
       Analog of build-docker-image job, but with a buildset registry.