Browse Source

Merge "docs: Elaborate on document layering in documentation"

Zuul 5 months ago
parent
commit
1d4cc81dfa

+ 7
- 2
doc/source/users/document-types.rst View File

@@ -111,9 +111,14 @@ correct schemas.
111 111
 
112 112
 .. _JSON schema: http://json-schema.org
113 113
 
114
+.. _layering-policy:
115
+
114 116
 LayeringPolicy
115 117
 ^^^^^^^^^^^^^^
116 118
 
119
+This document defines the strict order in which documents are layered together
120
+from their component parts.
121
+
117 122
 Only one ``LayeringPolicy`` document can exist within the system at any time.
118 123
 It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
119 124
 a different ``metadata.name`` than the existing document. If the names match,
@@ -127,8 +132,8 @@ it is treated as an update to the existing document.
127 132
   the new ``LayeringPolicy``.
128 133
 
129 134
 This document defines the strict order in which documents are merged together
130
-from their component parts. It should result in a validation error if a
131
-document refers to a layer not specified in the ``LayeringPolicy``.
135
+from their component parts. An error is raised if a document refers to a layer
136
+not specified in the ``LayeringPolicy``.
132 137
 
133 138
 Below is an example of a ``LayeringPolicy`` document:
134 139
 

+ 2
- 0
doc/source/users/documents.rst View File

@@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`,
34 34
 :ref:`revision-history` and :ref:`validation` should be reviewed for a more
35 35
 thorough understanding of each concept.
36 36
 
37
+.. _document-format:
38
+
37 39
 Document Format
38 40
 ---------------
39 41
 

+ 357
- 19
doc/source/users/layering.rst View File

@@ -23,38 +23,376 @@ Introduction
23 23
 ------------
24 24
 
25 25
 Layering provides a restricted data inheritance model intended to help reduce
26
-duplication in configuration. Documents with different ``schema``'s are never
27
-layered together (see the :ref:`substitution` section if you need to combine data
28
-from multiple types of documents).
26
+duplication in configuration. With layering, child documents can inherit
27
+data from parent documents. Through :ref:`layering-actions`, child documents
28
+can control exactly what they inherit from their parent. Document layering,
29
+conceptually speaking, works much like class inheritance: A child class
30
+inherits all variables and methods from its parent, but can elect to override
31
+its parent's functionality.
32
+
33
+Goals behind layering include:
34
+
35
+* model site deployment data hierarchically
36
+* lessen data duplication across site layers (as well as other conceptual
37
+  layers)
38
+
39
+Document Abstraction
40
+^^^^^^^^^^^^^^^^^^^^
41
+
42
+Layering works with :ref:`document-abstraction`: child documents can inherit
43
+from abstract as well as concrete parent documents.
44
+
45
+Pre-Conditions
46
+^^^^^^^^^^^^^^
47
+
48
+A document only has one parent, but its parent is computed dynamically using
49
+the :ref:`parent-selection` algorithm. That is, the notion of
50
+"multiple inheritance" **does not** apply to document layering.
51
+
52
+Documents with different ``schema`` values are never layered together (see the
53
+:ref:`substitution` section if you need to combine data from multiple types of
54
+documents).
55
+
56
+Document layering requires a :ref:`layering-policy` to exist in the revision
57
+whose documents will be layered together (rendered). An error will be issued
58
+otherwise.
59
+
60
+Terminology
61
+-----------
62
+
63
+.. note::
64
+
65
+  Whether a layer is "lower" or "higher" has entirely to do with its order of
66
+  initialization in a ``layerOrder`` and, by extension, its precedence in the
67
+  :ref:`parent-selection` algorithm described below.
68
+
69
+* Layer - A position in a hierarchy used to control :ref:`parent-selection` by
70
+  the :ref:`layering-algorithm`. It can be likened to a position in an
71
+  inheritance hierarchy, where ``object`` in Python can be likened to the
72
+  highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened
73
+  to the lowest layer in a ``layerOrder``.
74
+* Child - Meaningful only in a parent-child document relationship. A document
75
+  with a lower layer (but higher priority) than its parent, determined using
76
+  using :ref:`parent-selection`.
77
+* Parent - Meaningful only in a parent-child document relationship. A document
78
+  with a higher layer (but lower priority) than its child.
79
+* Layering Policy - A :ref:`control document <control-documents>` that defines
80
+  the strict ``layerOrder`` in which documents are layered together. See
81
+  :ref:`layering-policy` documentation for more information.
82
+* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the
83
+  :ref:`layering-policy` document. Establishes the layering hierarchy for a
84
+  set of layers in the system.
85
+* Layering Definition (``layeringDefinition``) - Metadata in each document for
86
+  controlling the following:
87
+
88
+  * ``layer``: the document layer itself
89
+  * ``parentSelector``: :ref:`parent-selection`
90
+  * ``abstract``: :ref:`document-abstraction`
91
+  * ``actions``: :ref:`layering-actions`
92
+
93
+* Parent Selector (``parentSelector``) - Key-value pairs or labels for
94
+  identifying the document's parent. Note that these key-value pairs are not
95
+  unique and that multiple documents can use them. All the key-value pairs
96
+  in the ``parentSelector`` must be found among the target parent's
97
+  ``metadata.labels``: this means that the ``parentSelector`` key-value pairs
98
+  must be a subset of the target parent's ``metadata.labels`` key-value
99
+  pairs. See :ref:`parent-selection` for further details.
100
+* Layering Actions (``actions``) - A list of actions that control what data
101
+  are inherited from the parent by the child. See :ref:`layering-actions`
102
+  for further details.
103
+
104
+.. _layering-algorithm:
105
+
106
+Algorithm
107
+---------
108
+
109
+Layering is applied at the bottommost layer of the ``layerOrder`` first and
110
+at the topmost layer of the ``layerOrder`` last, such that the "base" layers
111
+are processed first and the "leaf" layers are processed last. For each
112
+layer in the ``layerOrder``, the documents that correspond to that layer
113
+are retrieved. For each document retrieved, the ``layerOrder`` hierarchy
114
+is resolved using :ref:`parent-selection` to identify the parent document.
115
+Finally, the current document is layered with its parent using
116
+:ref:`layering-actions`.
117
+
118
+After layering is complete, the :ref:`substitution` algorithm is applied to the
119
+*current* document, if applicable.
120
+
121
+.. _layering-configuration:
122
+
123
+Layering Configuration
124
+----------------------
125
+
126
+Layering is configured in 2 places:
127
+
128
+#. The ``LayeringPolicy`` control document (described in
129
+   :ref:`layering-policy`), which defines the valid layers and their order of
130
+   precedence.
131
+#. In the ``metadata.layeringDefinition`` section of normal
132
+   (``metadata.schema=metadata/Document/v1``) documents. For more information
133
+   about document structure, reference :ref:`document-format`.
134
+
135
+An example ``layeringDefinition`` may look like::
136
+
137
+  layeringDefinition:
138
+    # Controls whether the document is abstract or concrete.
139
+    abstract: true
140
+    # A layer in the ``layerOrder``. Must be valid or else an error is raised.
141
+    layer: region
142
+    # Key-value pairs or labels for identifying the document's parent.
143
+    parentSelector:
144
+      required_key_a: required_label_a
145
+      required_key_b: required_label_b
146
+    # Actions which specify which data to add to the child document.
147
+    actions:
148
+      - method: merge
149
+        path: .path.to.merge.into.parent
150
+      - method: delete
151
+        path: .path.to.delete
152
+
153
+.. _layering-actions:
154
+
155
+Layering Actions
156
+----------------
157
+
158
+Introduction
159
+^^^^^^^^^^^^
160
+
161
+Layering actions allow child documents to modify data that is inherited from
162
+the parent. What if the child document should only inherit some of the parent
163
+data? No problem. A merge action can be performed, followed by ``delete``
164
+and ``replace`` actions to trim down on what should be inherited.
165
+
166
+Each layer action consists of an ``action`` and a ``path``. Whenever *any*
167
+action is specified, *all* the parent data is automatically inherited by the
168
+child document. The ``path`` specifies which data from the *child* document to
169
+**prioritize over** that of the parent document. Stated differently, all data
170
+from the parent is considered while *only* the *child* data at ``path`` is
171
+considered during an action. However, whenever a conflict occurs during an
172
+action, the *child* data takes priority over that of the parent.
173
+
174
+Layering actions are queued -- meaning that if a ``merge`` is
175
+specified before a ``replace`` then the ``merge`` will *necessarily* be
176
+applied before the ``replace``. For example, a ``merge`` followed by a
177
+``replace`` **is not necessarily** the same as a ``replace`` followed by a
178
+``merge``.
179
+
180
+Layering actions can be applied to primitives, lists and dictionaries alike.
181
+
182
+Action Types
183
+^^^^^^^^^^^^
184
+
185
+Supported actions are:
186
+
187
+* ``merge`` - "deep" merge child data and parent data into the child ``data``,
188
+  at the specified `JSONPath`_
189
+
190
+  .. note::
191
+
192
+    For conflicts between the child and parent data, the child document's
193
+    data is **always** prioritized. No other conflict resolution strategy for
194
+    this action currently exists.
195
+
196
+  ``merge`` behavior depends upon the data types getting merged. For objects
197
+  and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those
198
+  entities, after which Deckhand applies merge strategies (see below) to
199
+  combine merge child and parent data into the child document's ``data``
200
+  section.
201
+
202
+  **Merge Strategies**
203
+
204
+  Deckhand applies the following merge strategies for each data type:
205
+
206
+  * object: "Deep-merge" child and parent data together; conflicts are resolved
207
+    by prioritizing child data over parent data. "Deep-merge" means
208
+    recursively combining data for each key-value pair in both objects.
209
+  * array: The merge strategy involves:
210
+
211
+    * When using an index in the action ``path`` (e.g. ``a[0]``):
212
+
213
+      #. Copying the parent array into the child's ``data`` section at the
214
+         specified JSONPath.
215
+      #. Appending each child entry in the original child array into the parent
216
+         array. This behavior is synonymous with the ``extend`` list function
217
+         in Python.
218
+
219
+    * When not using an index in the action ``path`` (e.g. ``a``):
220
+
221
+      #. The child's array replaces the parent's array.
222
+  * primitives: Includes all other data types, except for ``null``. In this
223
+    case JSONPath resolution is impossible, so child data is prioritized over
224
+    that of the parent.
225
+
226
+  **Examples**
227
+
228
+  Given::
229
+
230
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
231
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
232
+
233
+  * When::
234
+
235
+      Merge Path: ``.``
236
+
237
+    Then::
238
+
239
+      Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
240
+
241
+      All data from parent is automatically considered, all data from child
242
+      is considered due to ``.`` (selects everything), then both merged.
243
+
244
+  * When::
245
+
246
+      Merge Path: ``.a``
247
+
248
+    Then::
249
+
250
+      Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
251
+
252
+      All data from parent is automatically considered, all data from child
253
+      at ``.a`` is considered, then both merged.
254
+
255
+  * When::
256
+
257
+      Merge Path: ``.b``
258
+
259
+    Then::
260
+
261
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
262
+
263
+      All data from parent is automatically considered, all data from child
264
+      at ``.b`` is considered, then both merged.
265
+
266
+  * When::
267
+
268
+      Merge Path: ``.c``
269
+
270
+    Then::
271
+
272
+      Error raised (``.c`` missing in child).
273
+
274
+* ``replace`` - overwrite existing data with child data at the specified
275
+  JSONPath.
29 276
 
30
-Layering is controlled in two places:
277
+  **Examples**
31 278
 
32
-1. The ``LayeringPolicy`` control document (described below), which defines the
33
-   valid layers and their order of precedence.
34
-2. In the ``metadata.layeringDefinition`` section of normal
35
-   (``metadata.schema=metadata/Document/v1``) documents.
279
+  Given::
36 280
 
37
-When rendering a particular document, you resolve the chain of parents upward
38
-through the layers, and begin working back down each layer rendering at each
39
-document in the chain.
281
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
282
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
40 283
 
41
-When rendering each layer, the parent document is used as the starting point,
42
-so the entire contents of the parent are brought forward.  Then the list of
43
-`actions` will be applied in order.  Supported actions are:
284
+  * When::
44 285
 
45
-* ``merge`` - "deep" merge child data at the specified path into the existing
46
-  data
47
-* ``replace`` - overwrite existing data with child data at the specified path
48
-* ``delete`` - remove the existing data at the specified path
286
+      Replace Path: ``.``
287
+
288
+    Then::
289
+
290
+      Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
291
+
292
+      All data from parent is automatically considered, but is replaced by all
293
+      data from child at ``.`` (selects everything), so replaces everything
294
+      in parent.
295
+
296
+  * When::
297
+
298
+      Replace Path: ``.a``
299
+
300
+    Then::
301
+
302
+      Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
303
+
304
+      All data from parent is automatically considered, but is replaced by all
305
+      data from child at ``.a``, so replaces all parent data at ``.a``.
306
+
307
+  * When::
308
+
309
+      Replace Path: ``.b``
310
+
311
+    Then::
312
+
313
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
314
+
315
+      All data from parent is automatically considered, but is replaced by all
316
+      data from child at ``.b``, so replaces all parent data at ``.b``.
317
+
318
+      While ``.b`` isn't in the parent, it only needs to exist in the child.
319
+      In this case, something (from the child) replaces nothing (from the
320
+      parent).
321
+
322
+  * When::
323
+
324
+      Replace Path: ``.c``
325
+
326
+    Then::
327
+
328
+      Error raised (``.c`` missing in child).
329
+
330
+* ``delete`` - remove the existing data at the specified JSONPath.
331
+
332
+  **Examples**
333
+
334
+  Given::
335
+
336
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
337
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
338
+
339
+  * When::
340
+
341
+      Delete Path: ``.``
342
+
343
+    Then::
344
+
345
+      Rendered Data: ``{}``
346
+
347
+      Note that deletion of everything results in an empty dictionary by
348
+      default.
349
+
350
+  * When::
351
+
352
+      Delete Path: ``.a``
353
+
354
+    Then::
355
+
356
+      Rendered Data: ``{'c': 9}``
357
+
358
+      All data from Parent Data at ``.a`` was deleted, rest copied over.
359
+
360
+  * When::
361
+
362
+      Delete Path: ``.c``
363
+
364
+    Then::
365
+
366
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
367
+
368
+      All data from Parent Data at ``.c`` was deleted, rest copied over.
369
+
370
+  * When::
371
+
372
+      Replace Path: ``.b``
373
+
374
+    Then::
375
+
376
+      Error raised (``.b`` missing in child).
49 377
 
50 378
 After actions are applied for a given layer, substitutions are applied (see
51
-the Substitution section for details).
379
+the :ref:`substitution` section for details).
380
+
381
+.. _JSONPath: http://goessner.net/articles/JsonPath/
52 382
 
53 383
 .. _parent-selection:
54 384
 
55 385
 Parent Selection
56 386
 ----------------
57 387
 
388
+Parent selection is performed dynamically. Unlike :ref:`substitution`,
389
+parent selection does not target a specific document using ``schema`` and
390
+``name`` identifiers. Rather, parent selection respects the ``layerOrder``,
391
+selecting the highest precedence parent in accordance with the algorithm that
392
+follows. This allows flexibility in parent selection: if a document's immediate
393
+parent is removed in a revision, then, if applicable, the grandparent (in the
394
+previous revision) can become the document's parent (in the latest revision).
395
+
58 396
 Selection of document parents is controlled by the ``parentSelector`` field and
59 397
 works as follows:
60 398
 

+ 7
- 2
tools/build-docs.sh View File

@@ -4,8 +4,13 @@
4 4
 # files. Must be run from root project directory.
5 5
 
6 6
 set -ex
7
+
8
+# Generate architectural diagrams.
9
+mkdir -p doc/source/images
10
+python -m plantuml doc/source/diagrams/*.uml
11
+mv doc/source/diagrams/*.png doc/source/images
12
+
13
+# Generate documentation.
7 14
 rm -rf doc/build
8 15
 rm -rf releasenotes/build
9 16
 sphinx-build -W -b html doc/source doc/build/html
10
-python -m plantuml doc/source/diagrams/*.uml
11
-mv doc/source/diagrams/*.png doc/source/images

Loading…
Cancel
Save