# Deckhand Design

## Purpose

Deckhand is a document-based configuration storage service built with
auditability and validation in mind.

## Essential Functionality

* layering - helps reduce duplication in configuration while maintaining
  auditability across many sites
* substitution - provides separation between secret data and other
  configuration data, while allowing a simple interface for clients
* revision history - improves auditability and enables services to provide
  functional validation of a well-defined collection of documents that are
  meant to operate together
* validation - allows services to implement and register different kinds of
  validations and report errors

## Documents

All configuration data is stored entirely as structured documents, for which
schemas must be registered.

### Document Format

The document format is modeled loosely after Kubernetes practices. The top
level of each document is a dictionary with 3 keys: `schema`, `metadata`, and
`data`.

* `schema` - Defines the name of the JSON schema to be used for validation.
  Must have the form: `<namespace>/<kind>/<version>`, where the meaning of
  each component is:
  * `namespace` - Identifies the owner of this type of document. The
    values `deckhand` and `metadata` are reserved for internal use.
  * `kind` - Identifies a type of configuration resource in the namespace.
  * `version` - Describe the version of this resource, e.g. "v1".
* `metadata` - Defines details that Deckhand will inspect and understand. There
  are multiple schemas for this section as discussed below. All the various
  types of metadata include a `name` field which must be unique for each
  document `schema`.
* `data` - Data to be validated by the schema described by the `schema`
  field. Deckhand only interacts with content here as instructed to do so by
  the `metadata` section. The form of this section is considered to be
  completely owned by the `namespace` in the `schema`.

#### Document Metadata

There are 3 supported kinds of document metadata. Documents with `Document`
metadata are the most common, and are used for normal configuration data.
Documents with `Control` metadata are used to customize the behavior of
Deckhand. Documents with `Tombstone` metadata are used to delete pre-existing
documents with either `Document` or `Control` metadata.

##### schema: metadata/Document/v1

This type of metadata allows the following metadata hierarchy:

* `name` - string, required - Unique within a revision for a given `schema`.
* `storagePolicy` - string, required - Either `cleartext` or `encrypted`. If
  `encyrpted` is specified, then the `data` section of the document will be
  stored in an secure backend (likely via OpenStack Barbican). `metadata` and
  `schema` fields are always stored in cleartext.
* `layeringDefinition` - dict, required - Specifies layering details. See the
  Layering section below for details.
  * `abstract` - boolean, required - An abstract document is not expected to
    pass schema validation after layering and substitution are applied.
    Non-abstract (concrete) documents are.
  * `layer` - string, required - References a layer in the `LayeringPolicy`
    control document.
  * `parentSelector` - labels, optional - Used to construct document chains for
    executing merges.
  * `actions` - list, optional - A sequence of actions to apply this documents
    data during the merge process.
    * `method` - string, required - How to layer this content.
    * `path` - string, required - What content in this document to layer onto
      parent content.
* `substitutions` - list, optional - A sequence of substitutions to apply. See
  the Substitutions section for additional details.
  * `dest` - dict, required - A description of the inserted content destination.
    * `path` - string, required - The JSON path where the data will be placed
      into the `data` section of this document.
    * `pattern` - string, optional - A regex to search for in the string
      specified at `path` in this document and replace with the source data.
  * `src` - dict, required - A description of the inserted content source.
    * `schema` - string, required - The `schema` of the source document.
    * `name` - string, required - The `metadata.name` of the source document.
    * `path` - string, required - The JSON path from which to extract data in
      the source document relative to its `data` section.

Here is a fictitious example of a complete document which illustrates all the
valid fields in the `metadata` section.

```yaml
---
schema: some-service/ResourceType/v1
metadata:
  schema: metadata/Document/v1
  name: unique-name-given-schema
  storagePolicy: cleartext
  labels:
    genesis: enabled
    master: enabled
  layeringDefinition:
    abstract: true
    layer: region
    parentSelector:
      required_key_a: required_label_a
      required_key_b: required_label_b
    actions:
      - method: merge
        path: .path.to.merge.into.parent
      - method: delete
        path: .path.to.delete
  substitutions:
    - dest:
        path: .substitution.target
      src:
        schema: another-service/SourceType/v1
        name: name-of-source-document
        path: .source.path
data:
  path:
    to:
      merge:
        into:
          parent:
            foo: bar
          ignored:  # Will not be part of the resultant document after layering.
            data: here
  substitution:
    target: null  # Paths do not need to exist to be specified as substitution destinations.
...
```

##### schema: metadata/Control/v1

This schema is the same as the `Document` schema, except it omits the
`storagePolicy`, `layeringDefinition`, and `substitutions` keys, as these
actions are not supported on `Control` documents.

The complete list of valid `Control` document kinds is specified below along
with descriptions of each document kind.

##### schema: metadata/Tombstone/v1

The only valid key in a `Tombstone` metadata section is `name`.  Additionally,
the top-level `data` section should be omitted.

### Layering

Layering provides a restricted data inheritance model intended to help reduce
duplication in configuration. Documents with different `schema`s are never
layered together (see the Substitution section if you need to combine data
from multiple types of documents).

Layering is controlled in two places:

1. The `LayeringPolicy` control document (described below), which defines the
   valid layers and their order of precedence.
2. In the `metadata.layeringDefinition` section of normal
   (`metadata.schema=metadata/Document/v1`) documents.

When rendering a particular document, you resolve the chain of parents upward
through the layers, and begin working back down each layer rendering at each
document in the chain.

When rendering each layer, the parent document is used as the starting point,
so the entire contents of the parent are brought forward.  Then the list of
`actions` will be applied in order.  Supported actions are:

* `merge` - "deep" merge child data at the specified path into the existing data
* `replace` - overwrite existing data with child data at the specified path
* `delete` - remove the existing data at the specified path

After actions are applied for a given layer, substitutions are applied (see
the Substitution section for details).

Selection of document parents is controlled by the `parentSelector` field and
works as follows. A given document, `C`, that specifies a `parentSelector`
will have exactly one parent, `P`. Document `P` will be the highest
precedence (i.e. part of the lowest layer defined in the `layerOrder` list
from the `LayeringPolicy`) document that has the labels indicated by the
`parentSelector` (and possibly additional labels) from the set of all
documents of the same `schema` as `C` that are in layers above the layer `C`
is in. For example, consider the following sample documents:

```yaml
---
schema: deckhand/LayeringPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: layering-policy
data:
  layerOrder:
    - global
    - region
    - site
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: global-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: global
data:
  a:
    x: 1
    y: 2
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: region-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: region
    parentSelector:
      key1: value1
    actions:
      - method: replace
        path: .a
data:
  a:
    z: 3
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: site-1234
  layeringDefinition:
    layer: site
    parentSelector:
      key1: value1
    actions:
      - method: merge
        path: .
data:
  b: 4
...
```

When rendering, the parent chosen for `site-1234` will be `region-1234`,
since it is the highest precedence document that matches the label selector
defined by `parentSelector`, and the parent chosen for `region-1234` will be
`global-1234` for the same reason. The rendered result for `site-1234` would
be:

```yaml
---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    z: 3
  b: 4
...
```

If `region-1234` were later removed, then the parent chosen for `site-1234`
would become `global-1234`, and the rendered result would become:

```yaml
---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    x: 1
    y: 2
  b: 4
...
```

<!-- TODO: Add figures for this example, with region present, have site point
with dotted line at global and indicate in caption (or something) that it's
selected for but ignored, because there's a higher-precedence layer to select
-->

### Substitution

Substitution is primarily designed as a mechanism for inserting secrets into
configuration documents, but works for unencrypted source documents as well.
Substitution is applied at each layer after all merge actions occur.

Concrete (non-abstract) documents can be used as a source of substitution
into other documents. This substitution is layer-independent, so given the 3
layer example above, which includes `global`, `region` and `site` layers, a
document in the `region` layer could insert data from a document in the
`site` layer.

Here is a sample set of documents demonstrating subistution:

```yaml
---
schema: deckhand/Certificate/v1
metadata:
  name: example-cert
  storagePolicy: cleartext
  layeringDefinition:
    layer: site
data: |
  CERTIFICATE DATA
---
schema: deckhand/CertificateKey/v1
metadata:
  name: example-key
  storagePolicy: encrypted
  layeringDefinition:
    layer: site
data: |
  KEY DATA
---
schema: deckhand/Passphrase/v1
metadata:
  name: example-password
  storagePolicy: encrypted
  layeringDefinition:
    layer: site
data: my-secret-password
---
schema: armada/Chart/v1
metadata:
  name: example-chart-01
  storagePolicy: cleartext
  layeringDefinition:
    layer: region
  substitutions:
    - dest:
        path: .chart.values.tls.certificate
      src:
        schema: deckhand/Certificate/v1
        name: example-cert
        path: .
    - dest:
        path: .chart.values.tls.key
      src:
        schema: deckhand/CertificateKey/v1
        name: example-key
        path: .
    - dest:
        path: .chart.values.some_url
        pattern: INSERT_[A-Z]+_HERE
      src:
        schema: deckhand/Passphrase/v1
        name: example-password
        path: .
data:
  chart:
    details:
      data: here
    values:
      some_url: http://admin:INSERT_PASSWORD_HERE@service-name:8080/v1
...
```

The rendered document will look like:

```yaml
---
schema: armada/Chart/v1
metadata:
  name: example-chart-01
  storagePolicy: cleartext
  layeringDefinition:
    layer: region
  substitutions:
    - dest:
        path: .chart.values.tls.certificate
      src:
        schema: deckhand/Certificate/v1
        name: example-cert
        path: .
    - dest:
        path: .chart.values.tls.key
      src:
        schema: deckhand/CertificateKey/v1
        name: example-key
        path: .
    - dest:
        path: .chart.values.some_url
        pattern: INSERT_[A-Z]+_HERE
      src:
        schema: deckhand/Passphrase/v1
        name: example-password
        path: .
data:
  chart:
    details:
      data: here
    values:
      some_url: http://admin:my-secret-password@service-name:8080/v1
      tls:
        certificate: |
          CERTIFICATE DATA
        key: |
          KEY DATA
...
```

### Control Documents

Control documents (documents which have `metadata.schema=metadata/Control/v1`),
are special, and are used to control the behavior of Deckhand at runtime.  Only
the following types of control documents are allowed.

#### DataSchema

`DataSchema` documents are used by various services to register new schemas
that Deckhand can use for validation. No `DataSchema` documents with names
beginning with `deckhand/` or `metadata/` are allowed.  Tme `metadata.name`
field of each `DataSchema` document specifies the top level `schema` that it
is used to validate.

The contents of its `data` key are expected to be the json schema definition
for the target document type from the target's top level `data` key down.

<!-- TODO: give valid, tiny schema as example -->

```yaml
---
schema: deckhand/DataSchema/v1  # This specifies the official JSON schema meta-schema.
metadata:
  schema: metadata/Control/v1
  name: promenade/Node/v1  # Specifies the documents to be used for validation.
  labels:
    application: promenade
data:  # Valid JSON Schema is expected here.
  $schema: http://blah
...
```

#### LayeringPolicy

Only one `LayeringPolicy` document can exist within the system at any time.
It is an error to attempt to insert a new `LayeringPolicy` document if it has
a different `metadata.name` than the existing document. If the names match,
it is treated as an update to the existing document.

This document defines the strict order in which documents are merged together
from their component parts. It should result in a validation error if a
document refers to a layer not specified in the `LayeringPolicy`.

```yaml
---
schema: deckhand/LayeringPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: layering-policy
data:
  layerOrder:
    - global
    - site-type
    - region
    - site
    - force
...
```

#### ValidationPolicy

Unlike `LayeringPolicy`, many `ValidationPolicy` documents are allowed. This
allows services to check whether a particular revision (described below) of
documents meets a configurable set of validations without having to know up
front the complete list of validations.

Each validation `name` specified here is a reference to data that is postable
by other services. Names beginning with `deckhand` are reserved for internal
use. See the Validation section below for more details.

Since validations may indicate interactions with external and changing
circumstances, an optional `expiresAfter` key may be specified for each
validation as an ISO8601 duration. If no `expiresAfter` is specified, a
successful validation does not expire. Note that expirations are specific to
the combination of `ValidationPolicy` and validation, not to each validation
by itself.

```yaml
---
schema: deckhand/ValidationPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: site-deploy-ready
data:
  validations:
    - name: deckhand-schema-validation
    - name: drydock-site-validation
      expiresAfter: P1W
    - name: promenade-site-validation
      expiresAfter: P1W
    - name: armada-deployability-validation
...
```

### Provided Utility Document Kinds

These are documents that use the `Document` metadata schema, but live in the
`deckhand` namespace.

#### Certificate

```yaml
---
schema: deckhand/Certificate/v1
metadata:
  schema: metadata/Document/v1
  name: application-api
  storagePolicy: cleartext
data: |-
  -----BEGIN CERTIFICATE-----
  MIIDYDCCAkigAwIBAgIUKG41PW4VtiphzASAMY4/3hL8OtAwDQYJKoZIhvcNAQEL
  ...snip...
  P3WT9CfFARnsw2nKjnglQcwKkKLYip0WY2wh3FE7nrQZP6xKNaSRlh6p2pCGwwwH
  HkvVwA==
  -----END CERTIFICATE-----
...
```

#### CertificateKey

```yaml
---
schema: deckhand/CertificateKey/v1
metadata:
  schema: metadata/Document/v1
  name: application-api
  storagePolicy: encrypted
data: |-
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpQIBAAKCAQEAx+m1+ao7uTVEs+I/Sie9YsXL0B9mOXFlzEdHX8P8x4nx78/T
  ...snip...
  Zf3ykIG8l71pIs4TGsPlnyeO6LzCWP5WRSh+BHnyXXjzx/uxMOpQ/6I=
  -----END RSA PRIVATE KEY-----
...
```

#### Passphrase

```yaml
---
schema: deckhand/Passphrase/v1
metadata:
  schema: metadata/Document/v1
  name: application-admin-password
  storagePolicy: encrypted
data: some-password
...
```

## Revision History

Documents will be ingested in batches which will be given a revision index.
This provides a common language for describing complex validations on sets of
documents.

Revisions can be thought of as commits in a linear git history, thus looking
at a revision includes all content from previous revisions.

## Validation

The validation system provides a unified approach to complex validations that
require coordination of multiple documents and business logic that resides in
consumer services.

Services can report success or failure of named validations for a given
revision. Those validations can then be referenced by many `ValidationPolicy`
control documents. The intended purpose use is to allow a simple mapping that
enables consuming services to be able to quickly check whether the
configuration in Deckhand is in a valid state for performing a specific
action.

### Deckhand-Provided Validations

In addition to allowing 3rd party services to report configurable validation
statuses, Deckhand provides a few internal validations which are made
available immediately upon document ingestion.

Here is a list of internal validations:

* `deckhand-document-schema-validation` - All concrete documents in the
  revision successfully pass their JSON schema validations. Will cause
  this to report an error.
* `deckhand-policy-validation` - All required policy documents are in-place,
  and existing documents conform to those policies.  E.g. if a 3rd party
  document specifies a `layer` that is not present in the layering policy,
  that will cause this validation to report an error.

## Access Control

Deckhand will use standard OpenStack Role Based Access Control using the
following actions:

- `read_cleartext_document` - Read unencrypted documents.
- `read_encrypted_document` - Read (including substitution and layering)
  encrypted documents.
- `read_revision` - Read details about revisions.
- `read_validation` - Read validation policy status, and validation results,
  including error messages.
- `write_cleartext_document` - Create, update or delete unencrypted documents.
- `write_encrypted_document` - Create, update or delete encrypted documents.
- `write_validation` - Write validation results.

## API

This API will only support YAML as a serialization format. Since the IETF
does not provide an official media type for YAML, this API will use
`application/x-yaml`.

This is a description of the `v1.0` API. Documented paths are considered
relative to `/api/v1.0`.

### POST `/documents`

Accepts a multi-document YAML body and creates a new revision which adds
those documents. Updates are detected based on exact match to an existing
document of `schema` + `metadata.name`. Documents are "deleted" by including
documents with the tombstone metadata schema, such as:

```yaml
---
schema: any-namespace/AnyKind/v1
metadata:
  schema: metadata/Tombstone/v1
  name: name-to-delete
...
```

This endpoint is the only way to add, update, and delete documents. This
triggers Deckhand's internal schema validations for all documents.

If no changes are detected, a new revision should not be created. This allows
services to periodically re-register their schemas without creating
unnecessary revisions.

This endpoint uses the `write_cleartext_document` and
`write_encrypted_document` actions.

### GET `/revisions/{revision_id}/documents`

Returns a multi-document YAML response containing all the documents matching
the filters specified via query string parameters. Returned documents will be
as originally posted with no substitutions or layering applied.

Supported query string parameters:

* `schema` - string, optional - The top-level `schema` field to select. This
  may be partially specified by section, e.g., `schema=promenade` would select all
  `kind` and `version` schemas owned by promenade, or `schema=promenade/Node`
  which would select all versions of `promenade/Node` documents. One may not
  partially specify the namespace or kind, so `schema=promenade/No` would not
  select `promenade/Node/v1` documents, and `schema=prom` would not select
  `promenade` documents.
* `metadata.name` - string, optional
* `metadata.layeringDefinition.abstract` - string, optional - Valid values are
  the "true" and "false".
* `metadata.layeringDefinition.layer` - string, optional - Only return documents from
  the specified layer.
* `metadata.label` - string, optional, repeatable - Uses the format
  `metadata.label=key=value`. Repeating this parameter indicates all
  requested labels must apply (AND not OR).

This endpoint uses the `read_cleartext_document` and
`read_encrypted_document` actions.

### GET `/revisions/{revision_id}/rendered-documents`

Returns a multi-document YAML of fully layered and substituted documents. No
abstract documents will be returned. This is the primary endpoint that
consumers will interact with for their configuration.

Valid query parameters are the same as for
`/revisions/{revision_id}/documents`, minus the paremters in
`metadata.layeringDetinition`, which are not supported.

This endpoint uses the `read_cleartext_document` and
`read_encrypted_document` actions.

### GET `/revisions`

Lists existing revisions and reports basic details including a summary of
validation status for each `deckhand/ValidationPolicy` that is part of that
revision.

Sample response:

```yaml
---
count: 7
next: https://deckhand/api/v1.0/revisions?limit=2&offset=2
prev: null
results:
  - id: 0
    url: https://deckhand/api/v1.0/revisions/0
    createdAt: 2017-07-14T21:23Z
    validationPolicies:
      site-deploy-validation:
        status: failed
  - id: 1
    url: https://deckhand/api/v1.0/revisions/1
    createdAt: 2017-07-16T01:15Z
    validationPolicies:
      site-deploy-validation:
        status: succeeded
...
```

This endpoint uses the `read_revision` action.

### GET `/revisions/{{revision_id}}`

Get a detailed description of a particular revision. The status of each
`ValidationPolicy` belonging to the revision is also included. Valid values
for the status of each validation policy are:

* `succeded` - All validations associated with the policy are `succeeded`.
* `failed` - Any validation associated with the policy has status `failed`,
  `expired` or `missing`.

Sample response:

```yaml
---
id: 0
url: https://deckhand/api/v1.0/revisions/0
createdAt: 2017-07-14T021:23Z
validationPolicies:
  site-deploy-validation:
    url: https://deckhand/api/v1.0/revisions/0/documents?schema=deckhand/ValidationPolicy/v1&name=site-deploy-validation
    status: failed
    validations:
      - name: deckhand-schema-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/deckhand-schema-validation/0
        status: success
      - name: drydock-site-validation
        status: missing
      - name: promenade-site-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/promenade-site-validation/0
        status: expired
      - name: armada-deployability-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/armada-deployability-validation/0
        status: failed
...
```

Validation status is always for the most recent entry for a given validation.
A status of `missing` indicates that no entries have been created. A status
of `expired` indicates that the validation had succeeded, but the
`expiresAfter` limit specified in the `ValidationPolicy` has been exceeded.

This endpoint uses the `read_revision` action.

### POST `/revisions/{{revision_id}}/validations/{{name}}`

Add the results of a validation for a particular revision.

An example `POST` request body indicating validation success:

```yaml
---
status: succeeded
validator:
  name: promenade
  version: 1.1.2
...
```

An example `POST` request indicating validation failure:

```http
POST /api/v1.0/revisions/3/validations/promenade-site-validation
Content-Type: application/x-yaml

---
status: failed
errors:
  - documents:
      - schema: promenade/Node/v1
        name: node-document-name
      - schema: promenade/Masters/v1
        name: kubernetes-masters
    message: Node has master role, but not included in cluster masters list.
validator:
  name: promenade
  version: 1.1.2
...
```

This endpoint uses the `write_validation` action.

### GET `/revisions/{{revision_id}}/validations`

Gets the list of validations which have reported for this revision.

Sample response:

```yaml
---
count: 2
next: null
prev: null
results:
  - name: deckhand-schema-validation
    url: https://deckhand/api/v1.0/revisions/4/validations/deckhand-schema-validation
    status: success
  - name: promenade-site-validation
    url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation
    status: failure
...
```

This endpoint uses the `read_validation` action.

### GET `/revisions/{{revision_id}}/validations/{{name}}`

Gets the list of validation entry summaries that have been posted.

Sample response:

```yaml
---
count: 1
next: null
prev: null
results:
  - id: 0
    url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/0/entries/0
    status: failure
...
```

This endpoint uses the `read_validation` action.

### GET `/revisions/{{revision_id}}/validations/{{name}}/entries/{{entry_id}}`

Gets the full details of a particular validation entry, including all posted
error details.

Sample response:

```yaml
---
name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
status: failure
createdAt: 2017-07-16T02:03Z
expiresAfter: null
expiresAt: null
errors:
  - documents:
      - schema: promenade/Node/v1
        name: node-document-name
      - schema: promenade/Masters/v1
        name: kubernetes-masters
    message: Node has master role, but not included in cluster masters list.
...
```

This endpoint uses the `read_validation` action.

### DELETE `/docuemnts/{{schema}}/{{name}}`

Delete the specified document.  This is equivalent to posting a tombstone for
the document.

This endpoint uses the `write_cleartext_document` and
`write_encrypted_document` actions.