docs: Elaborate on document layering in documentation

This patchset elaborates on document layering in the documentation
to provide much greater clarity into what layering is and its
associated concepts, including: layer, layer order, layering policy,
layering definition, document abstraction, parent selection,
layering actions, etc.

Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906

doc/source/users/document-types.rst

@@ -111,9 +111,14 @@ correct schemas.
 
 .. _JSON schema: http://json-schema.org
 
+.. _layering-policy:
+
 LayeringPolicy
 ^^^^^^^^^^^^^^
 
+This document defines the strict order in which documents are layered together
+from their component parts.
+
 Only one ``LayeringPolicy`` document can exist within the system at any time.
 It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
 a different ``metadata.name`` than the existing document. If the names match,
@@ -127,8 +132,8 @@ it is treated as an update to the existing document.
   the new ``LayeringPolicy``.
 
 This document defines the strict order in which documents are merged together
-from their component parts. It should result in a validation error if a
+from their component parts. An error is raised if a document refers to a layer
-document refers to a layer not specified in the ``LayeringPolicy``.
+not specified in the ``LayeringPolicy``.
 
 Below is an example of a ``LayeringPolicy`` document:
 

doc/source/users/documents.rst

@@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`,
 :ref:`revision-history` and :ref:`validation` should be reviewed for a more
 thorough understanding of each concept.
 
+.. _document-format:
+
 Document Format
 ---------------
 

doc/source/users/layering.rst

@@ -23,38 +23,376 @@ Introduction
 ------------
 
 Layering provides a restricted data inheritance model intended to help reduce
-duplication in configuration. Documents with different ``schema``'s are never
+duplication in configuration. With layering, child documents can inherit
-layered together (see the :ref:`substitution` section if you need to combine data
+data from parent documents. Through :ref:`layering-actions`, child documents
-from multiple types of documents).
+can control exactly what they inherit from their parent. Document layering,
+conceptually speaking, works much like class inheritance: A child class
+inherits all variables and methods from its parent, but can elect to override
+its parent's functionality.
 
-Layering is controlled in two places:
+Goals behind layering include:
 
-1. The ``LayeringPolicy`` control document (described below), which defines the
+* model site deployment data hierarchically
-   valid layers and their order of precedence.
+* lessen data duplication across site layers (as well as other conceptual
-2. In the ``metadata.layeringDefinition`` section of normal
+  layers)
-   (``metadata.schema=metadata/Document/v1``) documents.
 
-When rendering a particular document, you resolve the chain of parents upward
+Document Abstraction
-through the layers, and begin working back down each layer rendering at each
+^^^^^^^^^^^^^^^^^^^^
-document in the chain.
 
-When rendering each layer, the parent document is used as the starting point,
+Layering works with :ref:`document-abstraction`: child documents can inherit
-so the entire contents of the parent are brought forward.  Then the list of
+from abstract as well as concrete parent documents.
-`actions` will be applied in order.  Supported actions are:
 
-* ``merge`` - "deep" merge child data at the specified path into the existing
+Pre-Conditions
-  data
+^^^^^^^^^^^^^^
-* ``replace`` - overwrite existing data with child data at the specified path
+
-* ``delete`` - remove the existing data at the specified path
+A document only has one parent, but its parent is computed dynamically using
+the :ref:`parent-selection` algorithm. That is, the notion of
+"multiple inheritance" **does not** apply to document layering.
+
+Documents with different ``schema`` values are never layered together (see the
+:ref:`substitution` section if you need to combine data from multiple types of
+documents).
+
+Document layering requires a :ref:`layering-policy` to exist in the revision
+whose documents will be layered together (rendered). An error will be issued
+otherwise.
+
+Terminology
+-----------
+
+.. note::
+
+  Whether a layer is "lower" or "higher" has entirely to do with its order of
+  initialization in a ``layerOrder`` and, by extension, its precedence in the
+  :ref:`parent-selection` algorithm described below.
+
+* Layer - A position in a hierarchy used to control :ref:`parent-selection` by
+  the :ref:`layering-algorithm`. It can be likened to a position in an
+  inheritance hierarchy, where ``object`` in Python can be likened to the
+  highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened
+  to the lowest layer in a ``layerOrder``.
+* Child - Meaningful only in a parent-child document relationship. A document
+  with a lower layer (but higher priority) than its parent, determined using
+  using :ref:`parent-selection`.
+* Parent - Meaningful only in a parent-child document relationship. A document
+  with a higher layer (but lower priority) than its child.
+* Layering Policy - A :ref:`control document <control-documents>` that defines
+  the strict ``layerOrder`` in which documents are layered together. See
+  :ref:`layering-policy` documentation for more information.
+* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the
+  :ref:`layering-policy` document. Establishes the layering hierarchy for a
+  set of layers in the system.
+* Layering Definition (``layeringDefinition``) - Metadata in each document for
+  controlling the following:
+
+  * ``layer``: the document layer itself
+  * ``parentSelector``: :ref:`parent-selection`
+  * ``abstract``: :ref:`document-abstraction`
+  * ``actions``: :ref:`layering-actions`
+
+* Parent Selector (``parentSelector``) - Key-value pairs or labels for
+  identifying the document's parent. Note that these key-value pairs are not
+  unique and that multiple documents can use them. All the key-value pairs
+  in the ``parentSelector`` must be found among the target parent's
+  ``metadata.labels``: this means that the ``parentSelector`` key-value pairs
+  must be a subset of the target parent's ``metadata.labels`` key-value
+  pairs. See :ref:`parent-selection` for further details.
+* Layering Actions (``actions``) - A list of actions that control what data
+  are inherited from the parent by the child. See :ref:`layering-actions`
+  for further details.
+
+.. _layering-algorithm:
+
+Algorithm
+---------
+
+Layering is applied at the bottommost layer of the ``layerOrder`` first and
+at the topmost layer of the ``layerOrder`` last, such that the "base" layers
+are processed first and the "leaf" layers are processed last. For each
+layer in the ``layerOrder``, the documents that correspond to that layer
+are retrieved. For each document retrieved, the ``layerOrder`` hierarchy
+is resolved using :ref:`parent-selection` to identify the parent document.
+Finally, the current document is layered with its parent using
+:ref:`layering-actions`.
+
+After layering is complete, the :ref:`substitution` algorithm is applied to the
+*current* document, if applicable.
+
+.. _layering-configuration:
+
+Layering Configuration
+----------------------
+
+Layering is configured in 2 places:
+
+#. The ``LayeringPolicy`` control document (described in
+   :ref:`layering-policy`), which defines the valid layers and their order of
+   precedence.
+#. In the ``metadata.layeringDefinition`` section of normal
+   (``metadata.schema=metadata/Document/v1``) documents. For more information
+   about document structure, reference :ref:`document-format`.
+
+An example ``layeringDefinition`` may look like::
+
+  layeringDefinition:
+    # Controls whether the document is abstract or concrete.
+    abstract: true
+    # A layer in the ``layerOrder``. Must be valid or else an error is raised.
+    layer: region
+    # Key-value pairs or labels for identifying the document's parent.
+    parentSelector:
+      required_key_a: required_label_a
+      required_key_b: required_label_b
+    # Actions which specify which data to add to the child document.
+    actions:
+      - method: merge
+        path: .path.to.merge.into.parent
+      - method: delete
+        path: .path.to.delete
+
+.. _layering-actions:
+
+Layering Actions
+----------------
+
+Introduction
+^^^^^^^^^^^^
+
+Layering actions allow child documents to modify data that is inherited from
+the parent. What if the child document should only inherit some of the parent
+data? No problem. A merge action can be performed, followed by ``delete``
+and ``replace`` actions to trim down on what should be inherited.
+
+Each layer action consists of an ``action`` and a ``path``. Whenever *any*
+action is specified, *all* the parent data is automatically inherited by the
+child document. The ``path`` specifies which data from the *child* document to
+**prioritize over** that of the parent document. Stated differently, all data
+from the parent is considered while *only* the *child* data at ``path`` is
+considered during an action. However, whenever a conflict occurs during an
+action, the *child* data takes priority over that of the parent.
+
+Layering actions are queued -- meaning that if a ``merge`` is
+specified before a ``replace`` then the ``merge`` will *necessarily* be
+applied before the ``replace``. For example, a ``merge`` followed by a
+``replace`` **is not necessarily** the same as a ``replace`` followed by a
+``merge``.
+
+Layering actions can be applied to primitives, lists and dictionaries alike.
+
+Action Types
+^^^^^^^^^^^^
+
+Supported actions are:
+
+* ``merge`` - "deep" merge child data and parent data into the child ``data``,
+  at the specified `JSONPath`_
+
+  .. note::
+
+    For conflicts between the child and parent data, the child document's
+    data is **always** prioritized. No other conflict resolution strategy for
+    this action currently exists.
+
+  ``merge`` behavior depends upon the data types getting merged. For objects
+  and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those
+  entities, after which Deckhand applies merge strategies (see below) to
+  combine merge child and parent data into the child document's ``data``
+  section.
+
+  **Merge Strategies**
+
+  Deckhand applies the following merge strategies for each data type:
+
+  * object: "Deep-merge" child and parent data together; conflicts are resolved
+    by prioritizing child data over parent data. "Deep-merge" means
+    recursively combining data for each key-value pair in both objects.
+  * array: The merge strategy involves:
+
+    * When using an index in the action ``path`` (e.g. ``a[0]``):
+
+      #. Copying the parent array into the child's ``data`` section at the
+         specified JSONPath.
+      #. Appending each child entry in the original child array into the parent
+         array. This behavior is synonymous with the ``extend`` list function
+         in Python.
+
+    * When not using an index in the action ``path`` (e.g. ``a``):
+
+      #. The child's array replaces the parent's array.
+  * primitives: Includes all other data types, except for ``null``. In this
+    case JSONPath resolution is impossible, so child data is prioritized over
+    that of the parent.
+
+  **Examples**
+
+  Given::
+
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
+
+  * When::
+
+      Merge Path: ``.``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
+
+      All data from parent is automatically considered, all data from child
+      is considered due to ``.`` (selects everything), then both merged.
+
+  * When::
+
+      Merge Path: ``.a``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
+
+      All data from parent is automatically considered, all data from child
+      at ``.a`` is considered, then both merged.
+
+  * When::
+
+      Merge Path: ``.b``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
+
+      All data from parent is automatically considered, all data from child
+      at ``.b`` is considered, then both merged.
+
+  * When::
+
+      Merge Path: ``.c``
+
+    Then::
+
+      Error raised (``.c`` missing in child).
+
+* ``replace`` - overwrite existing data with child data at the specified
+  JSONPath.
+
+  **Examples**
+
+  Given::
+
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
+
+  * When::
+
+      Replace Path: ``.``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
+
+      All data from parent is automatically considered, but is replaced by all
+      data from child at ``.`` (selects everything), so replaces everything
+      in parent.
+
+  * When::
+
+      Replace Path: ``.a``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
+
+      All data from parent is automatically considered, but is replaced by all
+      data from child at ``.a``, so replaces all parent data at ``.a``.
+
+  * When::
+
+      Replace Path: ``.b``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
+
+      All data from parent is automatically considered, but is replaced by all
+      data from child at ``.b``, so replaces all parent data at ``.b``.
+
+      While ``.b`` isn't in the parent, it only needs to exist in the child.
+      In this case, something (from the child) replaces nothing (from the
+      parent).
+
+  * When::
+
+      Replace Path: ``.c``
+
+    Then::
+
+      Error raised (``.c`` missing in child).
+
+* ``delete`` - remove the existing data at the specified JSONPath.
+
+  **Examples**
+
+  Given::
+
+    Child Data:    ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
+    Parent Data:   ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
+
+  * When::
+
+      Delete Path: ``.``
+
+    Then::
+
+      Rendered Data: ``{}``
+
+      Note that deletion of everything results in an empty dictionary by
+      default.
+
+  * When::
+
+      Delete Path: ``.a``
+
+    Then::
+
+      Rendered Data: ``{'c': 9}``
+
+      All data from Parent Data at ``.a`` was deleted, rest copied over.
+
+  * When::
+
+      Delete Path: ``.c``
+
+    Then::
+
+      Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
+
+      All data from Parent Data at ``.c`` was deleted, rest copied over.
+
+  * When::
+
+      Replace Path: ``.b``
+
+    Then::
+
+      Error raised (``.b`` missing in child).
 
 After actions are applied for a given layer, substitutions are applied (see
-the Substitution section for details).
+the :ref:`substitution` section for details).
+
+.. _JSONPath: http://goessner.net/articles/JsonPath/
 
 .. _parent-selection:
 
 Parent Selection
 ----------------
 
+Parent selection is performed dynamically. Unlike :ref:`substitution`,
+parent selection does not target a specific document using ``schema`` and
+``name`` identifiers. Rather, parent selection respects the ``layerOrder``,
+selecting the highest precedence parent in accordance with the algorithm that
+follows. This allows flexibility in parent selection: if a document's immediate
+parent is removed in a revision, then, if applicable, the grandparent (in the
+previous revision) can become the document's parent (in the latest revision).
+
 Selection of document parents is controlled by the ``parentSelector`` field and
 works as follows:
 

tools/build-docs.sh

@@ -4,8 +4,13 @@
 # files. Must be run from root project directory.
 
 set -ex
+
+# Generate architectural diagrams.
+mkdir -p doc/source/images
+python -m plantuml doc/source/diagrams/*.uml
+mv doc/source/diagrams/*.png doc/source/images
+
+# Generate documentation.
 rm -rf doc/build
 rm -rf releasenotes/build
 sphinx-build -W -b html doc/source doc/build/html
-python -m plantuml doc/source/diagrams/*.uml
-mv doc/source/diagrams/*.png doc/source/images