Document replacement documentation

Follow up to [0] which includes document replacement
documentation. Includes documentation on the following
subjects:

  * document replacement (new)
  * document abstraction (new)
  * document parent selection (edit)

This PS also renames some docs files by replacing "_"
with "-" in files with compound names for consistency.

[0] https://review.gerrithub.io/#/c/403888/

Change-Id: I6c1ba1e77347c266a6a9d471c9d7a747b1cef6eb

docs/source/documents.rst

@@ -53,12 +53,18 @@ and ``data``.
 * ``metadata`` - Defines details that Deckhand will inspect and understand.
   There are multiple schemas for this section as discussed below. All the
   various types of metadata include a `name` field which must be unique for
-  each document `schema`.
+  each document ``schema``.
 * ``data`` - Data to be validated by the schema described by the ``schema``
   field. Deckhand only interacts with content here as instructed to do so by
   the ``metadata`` section. The form of this section is considered to be
   completely owned by the ``namespace`` in the ``schema``.
 
+Furthermore, documents are uniquely identified by the combination of:
+
+#. ``metadata.name``
+#. ``schema``
+#. ``metadata.layeringDefinition.layer``
+
 Below is a fictitious example of a complete document, which illustrates all the
 valid fields in the ``metadata`` section:
 
@@ -106,21 +112,22 @@ valid fields in the ``metadata`` section:
 Document Metadata
 ^^^^^^^^^^^^^^^^^
 
-There are 2 supported kinds of document metadata. Documents with `Document`
+There are 2 supported kinds of document metadata. Documents with ``Document``
 metadata are the most common, and are used for normal configuration data.
-Documents with `Control` metadata are used to customize the behavior of
+Documents with ``Control`` metadata are used to customize the behavior of
 Deckhand.
 
 schema: metadata/Document/v1
-""""""""""""""""""""""""""""
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This type of metadata allows the following metadata hierarchy:
 
-* ``name`` - string, required - Unique within a revision for a given ``schema``.
+* ``name`` - string, required - Unique within a revision for a given ``schema``
-* ``storagePolicy`` - string, required - Either ``cleartext`` or ``encrypted``. If
+  and ``metadata.layeringDefinition.layer``.
-  ``encyrpted`` is specified, then the ``data`` section of the document will be
+* ``storagePolicy`` - string, required - Either ``cleartext`` or ``encrypted``.
-  stored in an secure backend (likely via OpenStack Barbican). ``metadata`` and
+  If ``encyrpted`` is specified, then the ``data`` section of the document will
-  ``schema`` fields are always stored in cleartext.
+  be stored in a secure backend (likely via OpenStack Barbican). ``metadata``
+  and ``schema`` fields are always stored in cleartext.
 * ``layeringDefinition`` - dict, required - Specifies layering details. See the
   Layering section below for details.
 
@@ -156,7 +163,7 @@ This type of metadata allows the following metadata hierarchy:
 
 
 schema: metadata/Control/v1
-"""""""""""""""""""""""""""
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 This schema is the same as the ``Document`` schema, except it omits the
 ``storagePolicy``, ``layeringDefinition``, and ``substitutions`` keys, as these
@@ -164,3 +171,37 @@ actions are not supported on ``Control`` documents.
 
 The complete list of valid ``Control`` document kinds is specified below along
 with descriptions of each document kind.
+
+.. _document-abstraction:
+
+Document Abstraction
+--------------------
+
+Document abstraction can be compared to an abstract class in programming
+languages: The idea is to declare an abstract base class used for declaring
+common data to be overridden and customized by subclasses. In fact,
+this is the predominant use case for document abstraction: Defining base
+abstract documents that other concrete (non-abstract) documents can
+layer with.
+
+An abstract document is a document whose ``metadata.abstract`` property is
+True. A concrete document is a document whose ``metadata.abstract`` property
+is False. Concrete and non-abstract are terms that are used interchangeably.
+
+In Deckhand, document abstraction has certain implications:
+
+* An abstract document, like all other documents, will be persisted in
+  Deckhand's database and will be subjected to :ref:`revision-history`.
+* However, abstract documents are **not** returned by Deckhand's
+  ``rendered-documents`` endpoint: That is, rendered documents never include
+  abstract documents.
+* Concrete documents **can** layer with abstract documents -- and this is
+  encouraged.
+* Abstract documents **can** layer with other documents as well -- but unless
+  a concrete document layers with or substitutes from the resultant abstract
+  document, no meaningful data will be returned via rendering, as only
+  concrete documents are returned.
+* Likewise, abstract documents **can** substitute from other documents. The
+  same reasoning as the bullet point above applies.
+* However, abstract documents **cannot** be used as substitution sources.
+  Only concrete documents may be used as substitution sources.

docs/source/index.rst

@@ -40,12 +40,13 @@ User's Guide
 
    getting-started
    overview
+   revision-history
    documents
-   document_types
+   document-types
    validation
    substitution
    layering
-   revision_history
+   replacement
    api_ref
    api_client
    exceptions

docs/source/layering.rst

@@ -19,6 +19,9 @@
 Document Layering
 =================
 
+Introduction
+------------
+
 Layering provides a restricted data inheritance model intended to help reduce
 duplication in configuration. Documents with different ``schema``'s are never
 layered together (see the :ref:`substitution` section if you need to combine data
@@ -39,21 +42,60 @@ When rendering each layer, the parent document is used as the starting point,
 so the entire contents of the parent are brought forward.  Then the list of
 `actions` will be applied in order.  Supported actions are:
 
-* ``merge`` - "deep" merge child data at the specified path into the existing data
+* ``merge`` - "deep" merge child data at the specified path into the existing
+  data
 * ``replace`` - overwrite existing data with child data at the specified path
 * ``delete`` - remove the existing data at the specified path
 
 After actions are applied for a given layer, substitutions are applied (see
 the Substitution section for details).
 
+.. _parent-selection:
+
+Parent Selection
+----------------
+
 Selection of document parents is controlled by the ``parentSelector`` field and
-works as follows. A given document, ``C``, that specifies a ``parentSelector``
+works as follows:
-will have exactly one parent, ``P``. Document ``P`` will be the highest
+
-precedence (i.e. part of the lowest layer defined in the ``layerOrder`` list
+* A given document, ``C``, that specifies a ``parentSelector``, will have
-from the ``LayeringPolicy``) document that has the labels indicated by the
+  exactly one parent, ``P``. If comparing layering with inheritance,
-``parentSelector`` (and possibly additional labels) from the set of all
+  layering, then, does *not* allow multi-inheritance.
-documents of the same ``schema`` as ``C`` that are in layers above the layer ``C``
+* Both ``C`` and ``P`` must have the **same** ``schema``.
-is in. For example, consider the following sample documents:
+* Both ``C`` and ``P`` should have **different** ``metadata.name`` values
+  except in the case of :ref:`replacement`.
+* Document ``P`` will be the highest-precedence document whose
+  ``metadata.labels`` are a **superset** of document C's ``parentSelector``.
+  Where:
+
+  * Highest precendence means that ``P`` belongs to the lowest layer
+    defined in the ``layerOrder`` list from the ``LayeringPolicy`` which is
+    at least one level higher than the layer for ``C``. For example, if ``C``
+    has layer ``site``, then its parent ``P`` must at least have layer ``type``
+    or above in the following ``layerOrder``:
+
+    ::
+
+      ---
+      ...
+      layerOrder:
+        - global # Highest layer
+        - type
+        - site   # Lowest layer
+
+  * Superset means that ``P`` **at least** has all the labels in its
+    ``metadata.labels`` that child ``C`` references via its ``parentSelector``.
+    In other words, parent ``P`` can have more labels than ``C`` uses
+    to reference it, but ``C`` must at least have one matching label in its
+    ``parentSelector`` with ``P``.
+
+* Deckhand will select ``P`` if it belongs to the highest-precedence layer.
+  For example, if ``C`` belongs to layer ``site``, ``P`` belongs to layer
+  ``type``, and ``G`` belongs to layer ``global``, then Deckhand will use
+  ``P`` as the parent for ``C``. If ``P`` is non-existent, then ``G``
+  will be selected instead.
+
+For example, consider the following sample documents:
 
 .. code-block:: yaml
 

docs/source/overview.rst

@@ -141,3 +141,11 @@ and on demand provide the secrets to documents that need them. However,
 substitution can also apply to any data, not just secrets.
 
 For more information, see the :ref:`substitution` section.
+
+Replacement
+-----------
+
+Document replacement provides an advanced mechanism for reducing the overhead
+with data duplication across multiple documents.
+
+For more information, see the :ref:`replacement` section.

docs/source/replacement.rst

@@ -0,0 +1,244 @@
+..
+  Copyright 2018 AT&T Intellectual Property.
+  All Rights Reserved.
+
+  Licensed under the Apache License, Version 2.0 (the "License"); you may
+  not use this file except in compliance with the License. You may obtain
+  a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+  License for the specific language governing permissions and limitations
+  under the License.
+
+.. _replacement:
+
+Document Replacement
+====================
+
+.. note::
+
+  Document replacement is an advanced concept in Deckhand. This section assumes
+  that the reader already has an understanding of :ref:`layering` and
+  :ref:`substitution`.
+
+Document replacement, in the simplest terms, involves a *child* document
+replacing its *parent*. That is, the *entire* child document replaces its
+parent document. Replacement aims to lessen data duplication by taking
+advantage of :ref:`document-abstraction` and document layering patterns.
+
+.. todo::
+
+  Elaborate on these patterns in a separate section.
+
+Replacement introduces the ``replacement: true`` property underneath the
+top-level ``metadata`` section. This property is subject to certain
+preconditions, discussed in the `Requirements`_ section below.
+
+Replacement aims to replace specific values in a parent document via
+document replacement for particular sites, while allowing the same parent
+document to be consumed directly (layered with, substituted from) for
+completely different sites. This means that the same YAML template can be
+referenced from a global namespace by different site-level documents, and when
+necessary, specific sites can override the global defaults with specific
+overrides via document replacement. Effectively, this means that the same
+template can be referenced without having to duplicate all of its data, just to
+override a few values between the otherwise-exactly-the-same templates.
+
+Like abstract documents, documents that **are replaced** are not returned
+from Deckhand's ``rendered-documents`` endpoint. (Documents that
+**do replace** -- those with the ``replacement: true`` property -- are
+returned instead.)
+
+Requirements
+------------
+
+Document replacement has the following requirements:
+
+* Only a child document can replace its parent.
+* The child document must have the ``replacement: true`` property underneath
+  its ``metadata`` section.
+* The child document must be able to select the correct parent. For more
+  information on this, please reference the :ref:`parent-selection` section.
+* Additionally, the child document must have the **same** ``metadata.name``
+  and ``schema`` as its parent. Their ``metadata.layeringDefinition.layer``
+  must **differ**.
+
+The following result in validation errors:
+
+* A document with ``replacement: true`` doesn't have a parent.
+* A document with ``replacement: true`` doesn't have the same
+  ``metadata.name`` and ``schema`` as its parent.
+* A replacement document cannot iself be replaced. That is, only one level
+  of replacement is allowed.
+
+Note that each key in the examples below is *mandatory* and that the
+``parentSelector`` labels should be able to select the parent to be replaced.
+
+Document **replacer** (child):
+
+::
+
+  ---
+  # Note that the schema and metadata.name keys are the same as below.
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand
+    # The replacement: true key is mandatory.
+    replacement: true
+    layeringDefinition:
+      # Note that the layer differs from that of the parent below.
+      layer: N-1
+      # The key-value pairs underneath `parentSelector` must be compatible with
+      # key-value pairs underneath the `labels` section in the parent document
+      # below.
+      parentSelector:
+        selector: foo
+      actions:
+        - ...
+  data: ...
+
+Which replaces the document **replacee** (parent):
+
+::
+
+  ---
+  # Note that the schema and metadata.name keys are the same as above.
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand
+    labels:
+      selector: foo
+    layeringDefinition:
+      # Note that the layer differs from that of the child above.
+      layer: N
+  data: ...
+
+Why Replacement?
+----------------
+
+Layering without Replacement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Layering without replacement can introduce a lot of data duplication across
+documents. Take the following use case: Some sites need to be deployed with
+log debugging *enabled* and other sites need to be deployed with log debugging
+*disabled*.
+
+To achieve this, two top-layer documents can be created:
+
+::
+
+  ---
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand-1
+    layeringDefinition:
+      layer: global
+      ...
+  data:
+    debug: false
+    # Note that the data below can be arbitrarily long and complex.
+    ...
+
+And:
+
+::
+
+  ---
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand-2
+    layeringDefinition:
+      layer: global
+      ...
+  data:
+    debug: true
+    # Note that the data below can be arbitrarily long and complex.
+    ...
+
+However, what if the only thing that differs between the two documents is just
+``debug: true|false`` and every other value in both documents is precisely the
+same?
+
+Clearly, the pattern above leads to a lot of data duplication.
+
+Layering with Replacement
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using document replacement, the above duplication can be partially eliminated.
+For example:
+
+::
+
+  # Replacer (child document).
+  ---
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand
+    replacement: true
+    layeringDefinition:
+      layer: site
+      parentSelector:
+        selector: foo
+      actions:
+        - method: merge
+          path: .
+        - method: replace
+          path: .debug
+  data:
+    debug: True
+    ...
+
+And:
+
+::
+
+  # Replacee (parent document).
+  ---
+  schema: armada/Chart/v1
+  metadata:
+    name: ucp-deckhand
+    labels:
+      selector: foo
+    layeringDefinition:
+      layer: global
+      ...
+  data:
+    debug: false
+    ...
+
+In the case above, for sites that require ``debug: false``, only the
+global-level document should be included in the payload to Deckhand, along
+with all other documents required for site deployment.
+
+However, for sites that require ``debug: true``, both documents should be
+included in the payload to Deckhand, along with all other documents required
+for site deployment.
+
+Implications for Pegleg
+^^^^^^^^^^^^^^^^^^^^^^^
+
+In practice, when using `Pegleg`_, each document above can be placed in a
+separate file and Pegleg can either reference *only* the parent document
+if log debugging needs to be enabled or *both* documents if log debugging
+needs to be disabled. This pattern allows data duplication to be lessened.
+
+.. _Pegleg: http://pegleg.readthedocs.io/en/latest/
+
+How It Works
+------------
+
+Document replacement involves a child document replacing its parent. There
+are three fundamental cases that are handled:
+
+#. A child document replaces its parent. Only the child is returned.
+#. Same as (1), except that the parent document is used as a substitution
+   source. With replacement, the child is used as the substitution source
+   instead.
+#. Same as (2), except that the parent document is used as a layering
+   source (that is, yet another child document layers with the parent). With
+   replacement, the child is used as the layering source instead.