Browse Source

Document how to build a buildset registry

Change-Id: I57e6ace5f6731045b8043995b18835a499d2250c
master
James E. Blair 2 months ago
parent
commit
e620f0b74e
4 changed files with 380 additions and 0 deletions
  1. 2
    0
      doc/source/conf.py
  2. 375
    0
      doc/source/docker-image.rst
  3. 1
    0
      doc/source/index.rst
  4. 2
    0
      test-requirements.txt

+ 2
- 0
doc/source/conf.py View File

@@ -19,6 +19,8 @@
19 19
 extensions = [
20 20
     'sphinx.ext.autodoc',
21 21
     # 'sphinx.ext.intersphinx',
22
+    'sphinxcontrib.blockdiag',
23
+    'sphinxcontrib.seqdiag',
22 24
     'zuul_sphinx',
23 25
 ]
24 26
 

+ 375
- 0
doc/source/docker-image.rst View File

@@ -0,0 +1,375 @@
1
+Container Images
2
+================
3
+
4
+This repo has several jobs which can form the basis of a system
5
+supporting a full gating process for continuously deployed container
6
+images.  They can be used to build or test images which rely on other
7
+images using the full power of Zuul's speculative execution.
8
+
9
+In order to use these jobs to their full potential, the Zuul site
10
+administrator will need to run a simple but dedicated container image
11
+registry, and define local versions of the jobs to use it.  The
12
+following sections describe how to define those jobs and how the
13
+system is intended to work once the jobs are defined.
14
+
15
+Run an Intermediate Container Registry
16
+--------------------------------------
17
+
18
+A dedicated container registry is required for the use of these jobs.
19
+It is merely used to temporarily hold images so that they can be
20
+transferred between jobs running in different projects at different
21
+times.  It does not need to be publicly accessible or particularly
22
+robust.  If its backing storage fails and needs to be replaced, the
23
+only result is that some jobs running in Zuul may fail and may need to
24
+be re-run.  In this system, it is called the "intermediate registry"
25
+to distinguish it from other registry services.
26
+
27
+You may run the registry in whatever manner is appropriate for your
28
+site.  The following docker-compose file may be used as an example
29
+of a working deployment suitable for production:
30
+
31
+.. code-block:: yaml
32
+
33
+   services:
34
+     registry:
35
+       restart: always
36
+       image: registry:2
37
+       network_mode: host
38
+       environment:
39
+         REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
40
+         REGISTRY_HTTP_TLS_KEY: /certs/domain.key
41
+         REGISTRY_AUTH: htpasswd
42
+         REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
43
+         REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
44
+       volumes:
45
+         - /var/registry/data:/var/lib/registry
46
+         - /var/registry/certs:/certs
47
+         - /var/registry/auth:/auth
48
+
49
+You will need to provide the SSL certificate and key values, as well
50
+as the htpassword file with a user and password already present.
51
+
52
+Once that service is running, create the following four jobs in a
53
+Zuul config-project:
54
+
55
+.. _yoursite-buildset-registry:
56
+
57
+yoursite-buildset-registry
58
+~~~~~~~~~~~~~~~~~~~~~~~~~~
59
+
60
+This job is used to provide a temporary "buildset registry" to jobs
61
+running in your system; it communicates with the "intermediate"
62
+registry described above.
63
+
64
+.. code-block:: yaml
65
+   :caption: zuul.yaml
66
+
67
+   - secret:
68
+       name: yoursite-intermediate-registry
69
+       data:
70
+         host: insecure-ci-registry.example.org
71
+         port: 5000
72
+         username: zuul
73
+         password: !encrypted/pkcs1-oaep
74
+           - ...
75
+
76
+   - job:
77
+       name: yoursite-buildset-registry
78
+       pre-run: playbooks/buildset-registry/pre.yaml
79
+       run: playbooks/buildset-registry/run.yaml
80
+       post-run: playbooks/buildset-registry/post.yaml
81
+       secrets:
82
+         - secret: yoursite-intermediate-registry
83
+           name: intermediate_registry
84
+       requires: docker-image
85
+
86
+The credentials in the secret should match those you supplied when
87
+creating the intermediate registry.
88
+
89
+The ``requires: docker-image`` attribute means that whenever this job
90
+(or any jobs which inherit from it) run, Zuul will search ahead of the
91
+change in the dependency graph to find any jobs which produce
92
+docker-images and tell this job about them.  This allows the job to
93
+pull images from the intermediate registry into the buildset registry.
94
+
95
+.. code-block:: yaml
96
+   :caption: playbooks/buildset-registry/pre.yaml
97
+
98
+   - hosts: all
99
+     tasks:
100
+       - name: Install docker
101
+         include_role:
102
+           name: install-docker
103
+       - name: Run buildset registry (if not already running)
104
+         when: buildset_registry is not defined
105
+         include_role:
106
+           name: run-buildset-registry
107
+       - name: Use buildset registry
108
+         include_role:
109
+           name: use-buildset-registry
110
+
111
+   - hosts: localhost
112
+     roles:
113
+       - pull-from-intermediate-registry
114
+
115
+This playbook runs a buildset registry if one isn't already running.
116
+It returns the connection information back to Zuul in a variable
117
+called ``buildset_registry``.  Other jobs will use that to learn how
118
+to connect to the registry, and we can use that here to find out if
119
+one was already started in a previous job.  We will use that facility
120
+in the :ref:`yoursite-build-docker-image` job below.
121
+
122
+.. code-block:: yaml
123
+   :caption: playbooks/buildset-registry/run.yaml
124
+
125
+   - hosts: localhost
126
+     tasks:
127
+       - name: Pause the job
128
+         zuul_return:
129
+           data:
130
+             zuul:
131
+               pause: true
132
+
133
+The ``pause`` causes the job to wait until all jobs which depend on
134
+this one are completed.
135
+
136
+.. code-block:: yaml
137
+   :caption: playbooks/buildset-registry/post.yaml
138
+
139
+   - hosts: localhost
140
+     roles:
141
+       - push-to-intermediate-registry
142
+
143
+.. _yoursite-build-docker-image:
144
+
145
+yoursite-build-docker-image
146
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
147
+
148
+This job builds one or more docker images and interacts with the
149
+buildset and intermediate registries.
150
+
151
+.. code-block:: yaml
152
+   :caption: zuul.yaml
153
+
154
+   - job:
155
+       name: yoursite-build-docker-image
156
+       parent: yoursite-buildset-registry
157
+       run: playbooks/docker-image/run.yaml
158
+       provides: docker-image
159
+
160
+Note that the parent of this job is :ref:`yoursite-buildset-registry`.
161
+This means that a simple repo that only needs to support one image
162
+building job and doesn't have any other jobs which require a buildset
163
+registry can just add this job alone and it will run a buildset
164
+registry on the build host.  More complex scenarios would run the
165
+:ref:`yoursite-buildset-registry` job on its own and construct a job
166
+graph that depends on it.  Because the pre-run playbook in the
167
+buildset-registry job only runs a buildset registry if one isn't
168
+already running, it can be used for both cases.  And because the run
169
+playbook which pauses the job is overridden in this job, this job will
170
+not pause.
171
+
172
+.. code-block:: yaml
173
+   :caption: playbooks/docker-image/run.yaml
174
+
175
+   - hosts: all
176
+     roles:
177
+       - build-docker-image
178
+
179
+.. _yoursite-upload-docker-image:
180
+
181
+yoursite-upload-docker-image
182
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
183
+
184
+This job further builds on the :ref:`yoursite-build-docker-image` job
185
+and additionally uploads the image to Docker Hub.  Depending on the
186
+situation, you could encode the Docker Hub credentials into this job
187
+as a secret, or you could allow other users to provide them via the
188
+`pass-to-parent <https://zuul-ci.org/docs/zuul/user/config.html#attr-job.secrets.pass-to-parent>`_ feature of secrets.
189
+
190
+.. code-block:: yaml
191
+   :caption: zuul.yaml
192
+
193
+   - job:
194
+       name: yoursite-upload-docker-image
195
+       parent: yoursite-build-docker-image
196
+       post-run: playbooks/docker-image/upload.yaml
197
+
198
+.. code-block:: yaml
199
+   :caption: playbooks/docker-image/upload.yaml
200
+
201
+   - hosts: all
202
+     roles:
203
+       - upload-docker-image
204
+
205
+.. _yoursite-promote-docker-image:
206
+
207
+yoursite-promote-docker-image
208
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
209
+
210
+This job does nothing that the :zuul:job:`promote-docker-image` job in
211
+this repo doesn't already do, but since you created local versions of
212
+the other two jobs, you should make one of this as well for
213
+consistency.  If you chose to add Docker Hub credentials to the
214
+:ref:`yoursite-upload-docker-image` job, you should do that here as
215
+well.
216
+
217
+.. code-block:: yaml
218
+   :caption: zuul.yaml
219
+
220
+   - job:
221
+       name: yoursite-promote-docker-image
222
+       parent: promote-docker-image
223
+
224
+System Architecture
225
+-------------------
226
+
227
+Now that those jobs are defined, this section describes how they work
228
+together.
229
+
230
+There are a few key concepts to keep in mind:
231
+
232
+A *buildset* is a group of jobs all running on the same change.
233
+
234
+A *buildset registry* is a container image registry which is used to
235
+store speculatively built images for the use of jobs in a single
236
+buildset.  It holds the differences between the current state of the
237
+world and the future state if the change in question (and all of its
238
+dependent changes) were to merge.  It must be started by one of the
239
+jobs in a buildset, and it ceases to exist once that job is complete.
240
+
241
+An *intermediate registry* is a long-running registry that is used to
242
+store images created for unmerged changes for use by other unmerged
243
+changes.  It is not publicly accessible and is intended only to be
244
+used by Zuul in order to transfer artifacts from one buildset to
245
+another.
246
+
247
+With these concepts in mind, the jobs described above implement the
248
+following workflow for a single change:
249
+
250
+.. _buildset_image_transfer:
251
+
252
+.. seqdiag::
253
+   :caption: Buildset registry image transfer
254
+
255
+   seqdiag image_transfer {
256
+     Ireg [label="Intermediate\nRegistry"];
257
+     Breg [label="Buildset\nRegistry"];
258
+     Bjob [label="Image Build Job"];
259
+     Djob [label="Deployment Test Job"];
260
+
261
+     Ireg -> Breg [label='Images from previous changes'];
262
+     Breg -> Bjob [label='Images from previous changes'];
263
+     Breg <- Bjob [label='Current image'];
264
+     Ireg <- Breg [noactivate, label='Current image'];
265
+     Breg -> Djob [label='Current and previous images'];
266
+     Breg <- Djob [style=none];
267
+     Ireg <- Breg [style=none];
268
+   }
269
+
270
+The intermediate registry is always running and the buildset registry
271
+is started by a job running on a change.  The "Image Build" and
272
+"Deployment Test" jobs are example jobs which might be running on a
273
+change.  Essentially, these are image producer or consumer jobs
274
+respectively.
275
+
276
+There are two ways to use the jobs described above:
277
+
278
+A Repository with Producers and Consumers
279
+-----------------------------------------
280
+
281
+The first is in a repository where images are both produced and
282
+consumed.  In this case, we can expect that there will be at least one
283
+image build job, and at least one job which uses that image (for
284
+example, by performing a test deployment of the image).  In this case
285
+we need to construct a job graph with dependencies as follows:
286
+
287
+.. blockdiag::
288
+
289
+   blockdiag dependencies {
290
+     obr [label='yoursite-\nbuildset-registry'];
291
+     bi [label='build-image'];
292
+     ti [label='test-image'];
293
+
294
+     obr <- bi <- ti;
295
+   }
296
+
297
+The :ref:`yoursite-buildset-registry` job will run first and
298
+automatically start a buildset registry populated with images built
299
+from any changes which appear ahead of the current change.  It will
300
+then return its connection information to Zuul and pause and continue
301
+running until the completion of the build and test jobs.
302
+
303
+The build-image job should inherit from
304
+:ref:`yoursite-build-docker-image`, which will ensure that it is
305
+automatically configured to use the buildset registry.
306
+
307
+The test-image job is something that you will create yourself.  There
308
+is no standard way to test or deploy an image, that depends on your
309
+application.  However, there is one thing you will need to do in your
310
+job to take advantage of the buildset registry.  In a pre-run playbook,
311
+use the `use-buildset-registry
312
+<https://zuul-ci.org/docs/zuul-jobs/roles.html#role-use-buildset-registry>`_
313
+role:
314
+
315
+.. code-block:: yaml
316
+
317
+   - hosts: all
318
+     roles:
319
+       - use-buildset-registry
320
+
321
+That will configure the docker daemon on the host to use the buildset
322
+registry so that it will use the newly built version of any required
323
+images.
324
+
325
+A Repository with Only Producers
326
+--------------------------------
327
+
328
+The second way to use these jobs is in a repository where an image is
329
+merely built, but not deployed.  In this case, there are no consumers
330
+of the buildset registry other than the image build job, and so the
331
+registry can be run on the job itself.  In this case, you may omit the
332
+:ref:`yoursite-buildset-registry` job and run only the
333
+:ref:`yoursite-build-docker-image` job.
334
+
335
+Publishing an Image
336
+-------------------
337
+
338
+So far we've covered the image building process.  This system also
339
+provides two more jobs that are used in publishing images to Docker
340
+Hub.
341
+
342
+The :ref:`yoursite-upload-docker-image` job does everything the
343
+:ref:`yoursite-build-docker-image` job does, but it also uploads
344
+the built image to Docker Hub using an automatically-generated and
345
+temporary tag.  The "build" job is designed to be used in the
346
+*check* pipeline, while the "upload" job is designed to take its
347
+place in the *gate* pipeline.  By front-loading the upload to Docker
348
+Hub, we reduce the chance that a credential or network error will
349
+prevent us from publishing an image after a change lands.
350
+
351
+The :ref:`yoursite-promote-docker-image` job is designed to be
352
+used in the *promote* pipeline and simply re-tags the image on Docker
353
+Hub after the change lands.
354
+
355
+Keeping in mind that everything described above in
356
+:ref:`buildset_image_transfer` applies to the
357
+:ref:`yoursite-upload-docker-image` job, the following illustrates
358
+the additional tasks performed by the "upload" and "promote" jobs:
359
+
360
+.. seqdiag::
361
+
362
+   seqdiag image_transfer {
363
+     DH [activated, label="Docker Hub"];
364
+     Ujob [label="upload-image"];
365
+     Pjob [label="promote-image"];
366
+
367
+     DH -> Ujob [style=none];
368
+     DH <- Ujob [label='Current image with temporary tag'];
369
+     DH -> Pjob [label='Current image manifest with temporary tag',
370
+                 note='Only the manifest
371
+                       is transferred,
372
+                       not the actual
373
+                       image layers.'];
374
+     DH <- Pjob [label='Current image manifest with final tag'];
375
+   }

+ 1
- 0
doc/source/index.rst View File

@@ -7,6 +7,7 @@
7 7
    policy
8 8
    jobs
9 9
    roles
10
+   docker-image
10 11
 
11 12
 Indices and tables
12 13
 ==================

+ 2
- 0
test-requirements.txt View File

@@ -22,3 +22,5 @@ openstacksdk>=0.17.1
22 22
 requests
23 23
 requestsexceptions
24 24
 bs4
25
+sphinxcontrib-blockdiag>=1.1.0
26
+sphinxcontrib-seqdiag

Loading…
Cancel
Save