docs: Elaborate on document layering in documentation

This patchset elaborates on document layering in the documentation
to provide much greater clarity into what layering is and its
associated concepts, including: layer, layer order, layering policy,
layering definition, document abstraction, parent selection,
layering actions, etc.

Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906
This commit is contained in:
Felipe Monteiro 2018-06-27 16:37:42 -04:00
parent f9e4b5993f
commit 1ac9abb555
6 changed files with 373 additions and 23 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

View File

@ -111,9 +111,14 @@ correct schemas.
.. _JSON schema: http://json-schema.org
.. _layering-policy:
LayeringPolicy
^^^^^^^^^^^^^^
This document defines the strict order in which documents are layered together
from their component parts.
Only one ``LayeringPolicy`` document can exist within the system at any time.
It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
a different ``metadata.name`` than the existing document. If the names match,
@ -127,8 +132,8 @@ it is treated as an update to the existing document.
the new ``LayeringPolicy``.
This document defines the strict order in which documents are merged together
from their component parts. It should result in a validation error if a
document refers to a layer not specified in the ``LayeringPolicy``.
from their component parts. An error is raised if a document refers to a layer
not specified in the ``LayeringPolicy``.
Below is an example of a ``LayeringPolicy`` document:

View File

@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`,
:ref:`revision-history` and :ref:`validation` should be reviewed for a more
thorough understanding of each concept.
.. _document-format:
Document Format
---------------

View File

@ -23,38 +23,376 @@ Introduction
------------
Layering provides a restricted data inheritance model intended to help reduce
duplication in configuration. Documents with different ``schema``'s are never
layered together (see the :ref:`substitution` section if you need to combine data
from multiple types of documents).
duplication in configuration. With layering, child documents can inherit
data from parent documents. Through :ref:`layering-actions`, child documents
can control exactly what they inherit from their parent. Document layering,
conceptually speaking, works much like class inheritance: A child class
inherits all variables and methods from its parent, but can elect to override
its parent's functionality.
Layering is controlled in two places:
Goals behind layering include:
1. The ``LayeringPolicy`` control document (described below), which defines the
valid layers and their order of precedence.
2. In the ``metadata.layeringDefinition`` section of normal
(``metadata.schema=metadata/Document/v1``) documents.
* model site deployment data hierarchically
* lessen data duplication across site layers (as well as other conceptual
layers)
When rendering a particular document, you resolve the chain of parents upward
through the layers, and begin working back down each layer rendering at each
document in the chain.
Document Abstraction
^^^^^^^^^^^^^^^^^^^^
When rendering each layer, the parent document is used as the starting point,
so the entire contents of the parent are brought forward. Then the list of
`actions` will be applied in order. Supported actions are:
Layering works with :ref:`document-abstraction`: child documents can inherit
from abstract as well as concrete parent documents.
* ``merge`` - "deep" merge child data at the specified path into the existing
data
* ``replace`` - overwrite existing data with child data at the specified path
* ``delete`` - remove the existing data at the specified path
Pre-Conditions
^^^^^^^^^^^^^^
A document only has one parent, but its parent is computed dynamically using
the :ref:`parent-selection` algorithm. That is, the notion of
"multiple inheritance" **does not** apply to document layering.
Documents with different ``schema`` values are never layered together (see the
:ref:`substitution` section if you need to combine data from multiple types of
documents).
Document layering requires a :ref:`layering-policy` to exist in the revision
whose documents will be layered together (rendered). An error will be issued
otherwise.
Terminology
-----------
.. note::
Whether a layer is "lower" or "higher" has entirely to do with its order of
initialization in a ``layerOrder`` and, by extension, its precedence in the
:ref:`parent-selection` algorithm described below.
* Layer - A position in a hierarchy used to control :ref:`parent-selection` by
the :ref:`layering-algorithm`. It can be likened to a position in an
inheritance hierarchy, where ``object`` in Python can be likened to the
highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened
to the lowest layer in a ``layerOrder``.
* Child - Meaningful only in a parent-child document relationship. A document
with a lower layer (but higher priority) than its parent, determined using
using :ref:`parent-selection`.
* Parent - Meaningful only in a parent-child document relationship. A document
with a higher layer (but lower priority) than its child.
* Layering Policy - A :ref:`control document <control-documents>` that defines
the strict ``layerOrder`` in which documents are layered together. See
:ref:`layering-policy` documentation for more information.
* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the
:ref:`layering-policy` document. Establishes the layering hierarchy for a
set of layers in the system.
* Layering Definition (``layeringDefinition``) - Metadata in each document for
controlling the following:
* ``layer``: the document layer itself
* ``parentSelector``: :ref:`parent-selection`
* ``abstract``: :ref:`document-abstraction`
* ``actions``: :ref:`layering-actions`
* Parent Selector (``parentSelector``) - Key-value pairs or labels for
identifying the document's parent. Note that these key-value pairs are not
unique and that multiple documents can use them. All the key-value pairs
in the ``parentSelector`` must be found among the target parent's
``metadata.labels``: this means that the ``parentSelector`` key-value pairs
must be a subset of the target parent's ``metadata.labels`` key-value
pairs. See :ref:`parent-selection` for further details.
* Layering Actions (``actions``) - A list of actions that control what data
are inherited from the parent by the child. See :ref:`layering-actions`
for further details.
.. _layering-algorithm:
Algorithm
---------
Layering is applied at the bottommost layer of the ``layerOrder`` first and
at the topmost layer of the ``layerOrder`` last, such that the "base" layers
are processed first and the "leaf" layers are processed last. For each
layer in the ``layerOrder``, the documents that correspond to that layer
are retrieved. For each document retrieved, the ``layerOrder`` hierarchy
is resolved using :ref:`parent-selection` to identify the parent document.
Finally, the current document is layered with its parent using
:ref:`layering-actions`.
After layering is complete, the :ref:`substitution` algorithm is applied to the
*current* document, if applicable.
.. _layering-configuration:
Layering Configuration
----------------------
Layering is configured in 2 places:
#. The ``LayeringPolicy`` control document (described in
:ref:`layering-policy`), which defines the valid layers and their order of
precedence.
#. In the ``metadata.layeringDefinition`` section of normal
(``metadata.schema=metadata/Document/v1``) documents. For more information
about document structure, reference :ref:`document-format`.
An example ``layeringDefinition`` may look like::
layeringDefinition:
# Controls whether the document is abstract or concrete.
abstract: true
# A layer in the ``layerOrder``. Must be valid or else an error is raised.
layer: region
# Key-value pairs or labels for identifying the document's parent.
parentSelector:
required_key_a: required_label_a
required_key_b: required_label_b
# Actions which specify which data to add to the child document.
actions:
- method: merge
path: .path.to.merge.into.parent
- method: delete
path: .path.to.delete
.. _layering-actions:
Layering Actions
----------------
Introduction
^^^^^^^^^^^^
Layering actions allow child documents to modify data that is inherited from
the parent. What if the child document should only inherit some of the parent
data? No problem. A merge action can be performed, followed by ``delete``
and ``replace`` actions to trim down on what should be inherited.
Each layer action consists of an ``action`` and a ``path``. Whenever *any*
action is specified, *all* the parent data is automatically inherited by the
child document. The ``path`` specifies which data from the *child* document to
**prioritize over** that of the parent document. Stated differently, all data
from the parent is considered while *only* the *child* data at ``path`` is
considered during an action. However, whenever a conflict occurs during an
action, the *child* data takes priority over that of the parent.
Layering actions are queued -- meaning that if a ``merge`` is
specified before a ``replace`` then the ``merge`` will *necessarily* be
applied before the ``replace``. For example, a ``merge`` followed by a
``replace`` **is not necessarily** the same as a ``replace`` followed by a
``merge``.
Layering actions can be applied to primitives, lists and dictionaries alike.
Action Types
^^^^^^^^^^^^
Supported actions are:
* ``merge`` - "deep" merge child data and parent data into the child ``data``,
at the specified `JSONPath`_
.. note::
For conflicts between the child and parent data, the child document's
data is **always** prioritized. No other conflict resolution strategy for
this action currently exists.
``merge`` behavior depends upon the data types getting merged. For objects
and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those
entities, after which Deckhand applies merge strategies (see below) to
combine merge child and parent data into the child document's ``data``
section.
**Merge Strategies**
Deckhand applies the following merge strategies for each data type:
* object: "Deep-merge" child and parent data together; conflicts are resolved
by prioritizing child data over parent data. "Deep-merge" means
recursively combining data for each key-value pair in both objects.
* array: The merge strategy involves:
* When using an index in the action ``path`` (e.g. ``a[0]``):
#. Copying the parent array into the child's ``data`` section at the
specified JSONPath.
#. Appending each child entry in the original child array into the parent
array. This behavior is synonymous with the ``extend`` list function
in Python.
* When not using an index in the action ``path`` (e.g. ``a``):
#. The child's array replaces the parent's array.
* primitives: Includes all other data types, except for ``null``. In this
case JSONPath resolution is impossible, so child data is prioritized over
that of the parent.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Merge Path: ``.``
Then::
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, all data from child
is considered due to ``.`` (selects everything), then both merged.
* When::
Merge Path: ``.a``
Then::
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
All data from parent is automatically considered, all data from child
at ``.a`` is considered, then both merged.
* When::
Merge Path: ``.b``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, all data from child
at ``.b`` is considered, then both merged.
* When::
Merge Path: ``.c``
Then::
Error raised (``.c`` missing in child).
* ``replace`` - overwrite existing data with child data at the specified
JSONPath.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Replace Path: ``.``
Then::
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.`` (selects everything), so replaces everything
in parent.
* When::
Replace Path: ``.a``
Then::
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.a``, so replaces all parent data at ``.a``.
* When::
Replace Path: ``.b``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.b``, so replaces all parent data at ``.b``.
While ``.b`` isn't in the parent, it only needs to exist in the child.
In this case, something (from the child) replaces nothing (from the
parent).
* When::
Replace Path: ``.c``
Then::
Error raised (``.c`` missing in child).
* ``delete`` - remove the existing data at the specified JSONPath.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Delete Path: ``.``
Then::
Rendered Data: ``{}``
Note that deletion of everything results in an empty dictionary by
default.
* When::
Delete Path: ``.a``
Then::
Rendered Data: ``{'c': 9}``
All data from Parent Data at ``.a`` was deleted, rest copied over.
* When::
Delete Path: ``.c``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
All data from Parent Data at ``.c`` was deleted, rest copied over.
* When::
Replace Path: ``.b``
Then::
Error raised (``.b`` missing in child).
After actions are applied for a given layer, substitutions are applied (see
the Substitution section for details).
the :ref:`substitution` section for details).
.. _JSONPath: http://goessner.net/articles/JsonPath/
.. _parent-selection:
Parent Selection
----------------
Parent selection is performed dynamically. Unlike :ref:`substitution`,
parent selection does not target a specific document using ``schema`` and
``name`` identifiers. Rather, parent selection respects the ``layerOrder``,
selecting the highest precedence parent in accordance with the algorithm that
follows. This allows flexibility in parent selection: if a document's immediate
parent is removed in a revision, then, if applicable, the grandparent (in the
previous revision) can become the document's parent (in the latest revision).
Selection of document parents is controlled by the ``parentSelector`` field and
works as follows:

View File

@ -4,8 +4,13 @@
# files. Must be run from root project directory.
set -ex
# Generate architectural diagrams.
mkdir -p doc/source/images
python -m plantuml doc/source/diagrams/*.uml
mv doc/source/diagrams/*.png doc/source/images
# Generate documentation.
rm -rf doc/build
rm -rf releasenotes/build
sphinx-build -W -b html doc/source doc/build/html
python -m plantuml doc/source/diagrams/*.uml
mv doc/source/diagrams/*.png doc/source/images