airship/deckhand

Mark Burnett 4d302fa452 Clarify layering actions start from full parent data

2017-07-28 14:32:36 -05:00

26 KiB

Raw Blame History

Deckhand Design

Purpose

Deckhand is a document-based configuration storage service built with auditability and validation in mind.

Essential Functionality

layering - helps reduce duplication in configuration while maintaining auditability across many sites
substitution - provides separation between secret data and other configuration data, while allowing a simple interface for clients
revision history - improves auditability and enables services to provide functional validation of a well-defined collection of documents that are meant to operate together
validation - allows services to implement and register different kinds of validations and report errors

Documents

All configuration data is stored entirely as structured documents, for which schemas must be registered.

Document Format

The document format is modeled loosely after Kubernetes practices. The top level of each document is a dictionary with 3 keys: schema, metadata, and data.

schema - Defines the name of the JSON schema to be used for validation. Must have the form: <namespace>/<kind>/<version>, where the meaning of each component is:
- namespace - Identifies the owner of this type of document. The values deckhand and metadata are reserved for internal use.
- kind - Identifies a type of configuration resource in the namespace.
- version - Describe the version of this resource, e.g. "v1".
metadata - Defines details that Deckhand will inspect and understand. There are multiple schemas for this section as discussed below. All the various types of metadata include a name field which must be unique for each document schema.
data - Data to be validated by the schema described by the schema field. Deckhand only interacts with content here as instructed to do so by the metadata section. The form of this section is considered to be completely owned by the namespace in the schema.

Document Metadata

There are 3 supported kinds of document metadata. Documents with Document metadata are the most common, and are used for normal configuration data. Documents with Control metadata are used to customize the behavior of Deckhand. Documents with Tombstone metadata are used to delete pre-existing documents with either Document or Control metadata.

schema: metadata/Document/v1

This type of metadata allows the following metadata hierarchy:

name - string, required - Unique within a revision for a given schema.
storagePolicy - string, required - Either cleartext or encrypted. If encyrpted is specified, then the data section of the document will be stored in an secure backend (likely via OpenStack Barbican). metadata and schema fields are always stored in cleartext.
layeringDefinition - dict, required - Specifies layering details. See the Layering section below for details.
- abstract - boolean, required - An abstract document is not expected to pass schema validation after layering and substitution are applied. Non-abstract (concrete) documents are.
- layer - string, required - References a layer in the LayeringPolicy control document.
- parentSelector - labels, optional - Used to construct document chains for executing merges.
- actions - list, optional - A sequence of actions to apply this documents data during the merge process.
  - method - string, required - How to layer this content.
  - path - string, required - What content in this document to layer onto parent content.
substitutions - list, optional - A sequence of substitutions to apply. See the Substitutions section for additional details.
- dest - dict, required - A description of the inserted content destination.
  - path - string, required - The JSON path where the data will be placed into the data section of this document.
  - pattern - string, optional - A regex to search for in the string specified at path in this document and replace with the source data.
- src - dict, required - A description of the inserted content source.
  - schema - string, required - The schema of the source document.
  - name - string, required - The metadata.name of the source document.
  - path - string, required - The JSON path from which to extract data in the source document relative to its data section.

Here is a fictitious example of a complete document which illustrates all the valid fields in the metadata section.

---
schema: some-service/ResourceType/v1
metadata:
  schema: metadata/Document/v1
  name: unique-name-given-schema
  storagePolicy: cleartext
  labels:
    genesis: enabled
    master: enabled
  layeringDefinition:
    abstract: true
    layer: region
    parentSelector:
      required_key_a: required_label_a
      required_key_b: required_label_b
    actions:
      - method: merge
        path: .path.to.merge.into.parent
      - method: delete
        path: .path.to.delete
  substitutions:
    - dest:
        path: .substitution.target
      src:
        schema: another-service/SourceType/v1
        name: name-of-source-document
        path: .source.path
data:
  path:
    to:
      merge:
        into:
          parent:
            foo: bar
          ignored:  # Will not be part of the resultant document after layering.
            data: here
  substitution:
    target: null  # Paths do not need to exist to be specified as substitution destinations.
...

schema: metadata/Control/v1

This schema is the same as the Document schema, except it omits the storagePolicy, layeringDefinition, and substitutions keys, as these actions are not supported on Control documents.

The complete list of valid Control document kinds is specified below along with descriptions of each document kind.

schema: metadata/Tombstone/v1

The only valid key in a Tombstone metadata section is name. Additionally, the top-level data section should be omitted.

Layering

Layering provides a restricted data inheritance model intended to help reduce duplication in configuration. Documents with different schemas are never layered together (see the Substitution section if you need to combine data from multiple types of documents).

Layering is controlled in two places:

The LayeringPolicy control document (described below), which defines the valid layers and their order of precedence.
In the metadata.layeringDefinition section of normal (metadata.schema=metadata/Document/v1) documents.

When rendering a particular document, you resolve the chain of parents upward through the layers, and begin working back down each layer rendering at each document in the chain.

When rendering each layer, the parent document is used as the starting point, so the entire contents of the parent are brought forward. Then the list of actions will be applied in order. Supported actions are:

merge - "deep" merge child data at the specified path into the existing data
replace - overwrite existing data with child data at the specified path
delete - remove the existing data at the specified path

After actions are applied for a given layer, substitutions are applied (see the Substitution section for details).

Selection of document parents is controlled by the parentSelector field and works as follows. A given document, C, that specifies a parentSelector will have exactly one parent, P. Document P will be the highest precedence (i.e. part of the lowest layer defined in the layerOrder list from the LayeringPolicy) document that has the labels indicated by the parentSelector (and possibly additional labels) from the set of all documents of the same schema as C that are in layers above the layer C is in. For example, consider the following sample documents:

---
schema: deckhand/LayeringPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: layering-policy
data:
  layerOrder:
    - global
    - region
    - site
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: global-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: global
data:
  a:
    x: 1
    y: 2
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: region-1234
  labels:
    key1: value1
  layeringDefinition:
    abstract: true
    layer: region
    parentSelector:
      key1: value1
    actions:
      - method: replace
        path: .a
data:
  a:
    z: 3
---
schema: example/Kind/v1
metadata:
  schema: metadata/Document/v1
  name: site-1234
  layeringDefinition:
    layer: site
    parentSelector:
      key1: value1
    actions:
      - method: merge
        path: .
data:
  b: 4
...

When rendering, the parent chosen for site-1234 will be region-1234, since it is the highest precedence document that matches the label selector defined by parentSelector, and the parent chosen for region-1234 will be global-1234 for the same reason. The rendered result for site-1234 would be:

---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    z: 3
  b: 4
...

If region-1234 were later removed, then the parent chosen for site-1234 would become global-1234, and the rendered result would become:

---
schema: example/Kind/v1
metadata:
  name: site-1234
data:
  a:
    x: 1
    y: 2
  b: 4
...

Substitution

Substitution is primarily designed as a mechanism for inserting secrets into configuration documents, but works for unencrypted source documents as well. Substitution is applied at each layer after all merge actions occur.

Concrete (non-abstract) documents can be used as a source of substitution into other documents. This substitution is layer-independent, so given the 3 layer example above, which includes global, region and site layers, a document in the region layer could insert data from a document in the site layer.

Here is a sample set of documents demonstrating subistution:

---
schema: deckhand/Certificate/v1
metadata:
  name: example-cert
  storagePolicy: cleartext
  layeringDefinition:
    layer: site
data: |
  CERTIFICATE DATA  
---
schema: deckhand/CertificateKey/v1
metadata:
  name: example-key
  storagePolicy: encrypted
  layeringDefinition:
    layer: site
data: |
  KEY DATA  
---
schema: deckhand/Passphrase/v1
metadata:
  name: example-password
  storagePolicy: encrypted
  layeringDefinition:
    layer: site
data: my-secret-password
---
schema: armada/Chart/v1
metadata:
  name: example-chart-01
  storagePolicy: cleartext
  layeringDefinition:
    layer: region
  substitutions:
    - dest:
        path: .chart.values.tls.certificate
      src:
        schema: deckhand/Certificate/v1
        name: example-cert
        path: .
    - dest:
        path: .chart.values.tls.key
      src:
        schema: deckhand/CertificateKey/v1
        name: example-key
        path: .
    - dest:
        path: .chart.values.some_url
        pattern: INSERT_[A-Z]+_HERE
      src:
        schema: deckhand/Passphrase/v1
        name: example-password
        path: .
data:
  chart:
    details:
      data: here
    values:
      some_url: http://admin:INSERT_PASSWORD_HERE@service-name:8080/v1
...

The rendered document will look like:

---
schema: armada/Chart/v1
metadata:
  name: example-chart-01
  storagePolicy: cleartext
  layeringDefinition:
    layer: region
  substitutions:
    - dest:
        path: .chart.values.tls.certificate
      src:
        schema: deckhand/Certificate/v1
        name: example-cert
        path: .
    - dest:
        path: .chart.values.tls.key
      src:
        schema: deckhand/CertificateKey/v1
        name: example-key
        path: .
    - dest:
        path: .chart.values.some_url
        pattern: INSERT_[A-Z]+_HERE
      src:
        schema: deckhand/Passphrase/v1
        name: example-password
        path: .
data:
  chart:
    details:
      data: here
    values:
      some_url: http://admin:my-secret-password@service-name:8080/v1
      tls:
        certificate: |
          CERTIFICATE DATA          
        key: |
          KEY DATA          
...

Control Documents

Control documents (documents which have metadata.schema=metadata/Control/v1), are special, and are used to control the behavior of Deckhand at runtime. Only the following types of control documents are allowed.

DataSchema

DataSchema documents are used by various services to register new schemas that Deckhand can use for validation. No DataSchema documents with names beginning with deckhand/ or metadata/ are allowed. Tme metadata.name field of each DataSchema document specifies the top level schema that it is used to validate.

The contents of its data key are expected to be the json schema definition for the target document type from the target's top level data key down.

---
schema: deckhand/DataSchema/v1  # This specifies the official JSON schema meta-schema.
metadata:
  schema: metadata/Control/v1
  name: promenade/Node/v1  # Specifies the documents to be used for validation.
  labels:
    application: promenade
data:  # Valid JSON Schema is expected here.
  $schema: http://blah
...

LayeringPolicy

Only one LayeringPolicy document can exist within the system at any time. It is an error to attempt to insert a new LayeringPolicy document if it has a different metadata.name than the existing document. If the names match, it is treated as an update to the existing document.

This document defines the strict order in which documents are merged together from their component parts. It should result in a validation error if a document refers to a layer not specified in the LayeringPolicy.

---
schema: deckhand/LayeringPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: layering-policy
data:
  layerOrder:
    - global
    - site-type
    - region
    - site
    - force
...

ValidationPolicy

Unlike LayeringPolicy, many ValidationPolicy documents are allowed. This allows services to check whether a particular revision (described below) of documents meets a configurable set of validations without having to know up front the complete list of validations.

Each validation name specified here is a reference to data that is postable by other services. Names beginning with deckhand are reserved for internal use. See the Validation section below for more details.

Since validations may indicate interactions with external and changing circumstances, an optional expiresAfter key may be specified for each validation as an ISO8601 duration. If no expiresAfter is specified, a successful validation does not expire. Note that expirations are specific to the combination of ValidationPolicy and validation, not to each validation by itself.

---
schema: deckhand/ValidationPolicy/v1
metadata:
  schema: metadata/Control/v1
  name: site-deploy-ready
data:
  validations:
    - name: deckhand-schema-validation
    - name: drydock-site-validation
      expiresAfter: P1W
    - name: promenade-site-validation
      expiresAfter: P1W
    - name: armada-deployability-validation
...

Provided Utility Document Kinds

These are documents that use the Document metadata schema, but live in the deckhand namespace.

Certificate

---
schema: deckhand/Certificate/v1
metadata:
  schema: metadata/Document/v1
  name: application-api
  storagePolicy: cleartext
data: |-
  -----BEGIN CERTIFICATE-----
  MIIDYDCCAkigAwIBAgIUKG41PW4VtiphzASAMY4/3hL8OtAwDQYJKoZIhvcNAQEL
  ...snip...
  P3WT9CfFARnsw2nKjnglQcwKkKLYip0WY2wh3FE7nrQZP6xKNaSRlh6p2pCGwwwH
  HkvVwA==
  -----END CERTIFICATE-----  
...

CertificateKey

---
schema: deckhand/CertificateKey/v1
metadata:
  schema: metadata/Document/v1
  name: application-api
  storagePolicy: encrypted
data: |-
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpQIBAAKCAQEAx+m1+ao7uTVEs+I/Sie9YsXL0B9mOXFlzEdHX8P8x4nx78/T
  ...snip...
  Zf3ykIG8l71pIs4TGsPlnyeO6LzCWP5WRSh+BHnyXXjzx/uxMOpQ/6I=
  -----END RSA PRIVATE KEY-----  
...

Passphrase

---
schema: deckhand/Passphrase/v1
metadata:
  schema: metadata/Document/v1
  name: application-admin-password
  storagePolicy: encrypted
data: some-password
...

Revision History

Documents will be ingested in batches which will be given a revision index. This provides a common language for describing complex validations on sets of documents.

Revisions can be thought of as commits in a linear git history, thus looking at a revision includes all content from previous revisions.

Validation

The validation system provides a unified approach to complex validations that require coordination of multiple documents and business logic that resides in consumer services.

Services can report success or failure of named validations for a given revision. Those validations can then be referenced by many ValidationPolicy control documents. The intended purpose use is to allow a simple mapping that enables consuming services to be able to quickly check whether the configuration in Deckhand is in a valid state for performing a specific action.

Deckhand-Provided Validations

In addition to allowing 3rd party services to report configurable validation statuses, Deckhand provides a few internal validations which are made available immediately upon document ingestion.

Here is a list of internal validations:

deckhand-document-schema-validation - All concrete documents in the revision successfully pass their JSON schema validations. Will cause this to report an error.
deckhand-policy-validation - All required policy documents are in-place, and existing documents conform to those policies. E.g. if a 3rd party document specifies a layer that is not present in the layering policy, that will cause this validation to report an error.

Access Control

Deckhand will use standard OpenStack Role Based Access Control using the following actions:

read_cleartext_document - Read unencrypted documents.
read_encrypted_document - Read (including substitution and layering) encrypted documents.
read_revision - Read details about revisions.
read_validation - Read validation policy status, and validation results, including error messages.
write_cleartext_document - Create, update or delete unencrypted documents.
write_encrypted_document - Create, update or delete encrypted documents.
write_validation - Write validation results.

API

This API will only support YAML as a serialization format. Since the IETF does not provide an official media type for YAML, this API will use application/x-yaml.

This is a description of the v1.0 API. Documented paths are considered relative to /api/v1.0.

POST `/documents`

Accepts a multi-document YAML body and creates a new revision which adds those documents. Updates are detected based on exact match to an existing document of schema + metadata.name. Documents are "deleted" by including documents with the tombstone metadata schema, such as:

---
schema: any-namespace/AnyKind/v1
metadata:
  schema: metadata/Tombstone/v1
  name: name-to-delete
...

This endpoint is the only way to add, update, and delete documents. This triggers Deckhand's internal schema validations for all documents.

If no changes are detected, a new revision should not be created. This allows services to periodically re-register their schemas without creating unnecessary revisions.

This endpoint uses the write_cleartext_document and write_encrypted_document actions.

GET `/revisions/{revision_id}/documents`

Returns a multi-document YAML response containing all the documents matching the filters specified via query string parameters. Returned documents will be as originally posted with no substitutions or layering applied.

Supported query string parameters:

schema - string, optional - The top-level schema field to select. This may be partially specified by section, e.g., schema=promenade would select all kind and version schemas owned by promenade, or schema=promenade/Node which would select all versions of promenade/Node documents. One may not partially specify the namespace or kind, so schema=promenade/No would not select promenade/Node/v1 documents, and schema=prom would not select promenade documents.
metadata.name - string, optional
metadata.layeringDefinition.abstract - string, optional - Valid values are the "true" and "false".
metadata.layeringDefinition.layer - string, optional - Only return documents from the specified layer.
metadata.label - string, optional, repeatable - Uses the format metadata.label=key=value. Repeating this parameter indicates all requested labels must apply (AND not OR).

This endpoint uses the read_cleartext_document and read_encrypted_document actions.

GET `/revisions/{revision_id}/rendered-documents`

Returns a multi-document YAML of fully layered and substituted documents. No abstract documents will be returned. This is the primary endpoint that consumers will interact with for their configuration.

Valid query parameters are the same as for /revisions/{revision_id}/documents, minus the paremters in metadata.layeringDetinition, which are not supported.

This endpoint uses the read_cleartext_document and read_encrypted_document actions.

GET `/revisions`

Lists existing revisions and reports basic details including a summary of validation status for each deckhand/ValidationPolicy that is part of that revision.

Sample response:

---
count: 7
next: https://deckhand/api/v1.0/revisions?limit=2&offset=2
prev: null
results:
  - id: 0
    url: https://deckhand/api/v1.0/revisions/0
    createdAt: 2017-07-14T21:23Z
    validationPolicies:
      site-deploy-validation:
        status: failed
  - id: 1
    url: https://deckhand/api/v1.0/revisions/1
    createdAt: 2017-07-16T01:15Z
    validationPolicies:
      site-deploy-validation:
        status: succeeded
...

This endpoint uses the read_revision action.

GET `/revisions/{{revision_id}}`

Get a detailed description of a particular revision. The status of each ValidationPolicy belonging to the revision is also included. Valid values for the status of each validation policy are:

succeded - All validations associated with the policy are succeeded.
failed - Any validation associated with the policy has status failed, expired or missing.

Sample response:

---
id: 0
url: https://deckhand/api/v1.0/revisions/0
createdAt: 2017-07-14T021:23Z
validationPolicies:
  site-deploy-validation:
    url: https://deckhand/api/v1.0/revisions/0/documents?schema=deckhand/ValidationPolicy/v1&name=site-deploy-validation
    status: failed
    validations:
      - name: deckhand-schema-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/deckhand-schema-validation/0
        status: success
      - name: drydock-site-validation
        status: missing
      - name: promenade-site-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/promenade-site-validation/0
        status: expired
      - name: armada-deployability-validation
        url: https://deckhand/api/v1.0/revisions/0/validations/armada-deployability-validation/0
        status: failed
...

Validation status is always for the most recent entry for a given validation. A status of missing indicates that no entries have been created. A status of expired indicates that the validation had succeeded, but the expiresAfter limit specified in the ValidationPolicy has been exceeded.

This endpoint uses the read_revision action.

POST `/revisions/{{revision_id}}/validations/{{name}}`

Add the results of a validation for a particular revision.

An example POST request body indicating validation success:

---
status: succeeded
validator:
  name: promenade
  version: 1.1.2
...

An example POST request indicating validation failure:

POST /api/v1.0/revisions/3/validations/promenade-site-validation
Content-Type: application/x-yaml

---
status: failed
errors:
  - documents:
      - schema: promenade/Node/v1
        name: node-document-name
      - schema: promenade/Masters/v1
        name: kubernetes-masters
    message: Node has master role, but not included in cluster masters list.
validator:
  name: promenade
  version: 1.1.2
...

This endpoint uses the write_validation action.

GET `/revisions/{{revision_id}}/validations`

Gets the list of validations which have reported for this revision.

Sample response:

---
count: 2
next: null
prev: null
results:
  - name: deckhand-schema-validation
    url: https://deckhand/api/v1.0/revisions/4/validations/deckhand-schema-validation
    status: success
  - name: promenade-site-validation
    url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation
    status: failure
...

This endpoint uses the read_validation action.

GET `/revisions/{{revision_id}}/validations/{{name}}`

Gets the list of validation entry summaries that have been posted.

Sample response:

---
count: 1
next: null
prev: null
results:
  - id: 0
    url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/0/entries/0
    status: failure
...

This endpoint uses the read_validation action.

GET `/revisions/{{revision_id}}/validations/{{name}}/entries/{{entry_id}}`

Gets the full details of a particular validation entry, including all posted error details.

Sample response:

---
name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
status: failure
createdAt: 2017-07-16T02:03Z
expiresAfter: null
expiresAt: null
errors:
  - documents:
      - schema: promenade/Node/v1
        name: node-document-name
      - schema: promenade/Masters/v1
        name: kubernetes-masters
    message: Node has master role, but not included in cluster masters list.
...

This endpoint uses the read_validation action.

DELETE `/docuemnts/{{schema}}/{{name}}`

Delete the specified document. This is equivalent to posting a tombstone for the document.

This endpoint uses the write_cleartext_document and write_encrypted_document actions.

26 KiB Raw Blame History

Deckhand Design

Purpose

Essential Functionality

Documents

Document Format

Document Metadata

schema: metadata/Document/v1

schema: metadata/Control/v1

schema: metadata/Tombstone/v1

Layering

Substitution

Control Documents

DataSchema

LayeringPolicy

ValidationPolicy

Provided Utility Document Kinds

Certificate

CertificateKey

Passphrase

Revision History

Validation

Deckhand-Provided Validations

Access Control

API

POST /documents

GET /revisions/{revision_id}/documents

GET /revisions/{revision_id}/rendered-documents

GET /revisions

GET /revisions/{{revision_id}}

POST /revisions/{{revision_id}}/validations/{{name}}

GET /revisions/{{revision_id}}/validations

GET /revisions/{{revision_id}}/validations/{{name}}

GET /revisions/{{revision_id}}/validations/{{name}}/entries/{{entry_id}}

DELETE /docuemnts/{{schema}}/{{name}}

26 KiB

Raw Blame History

POST `/documents`

GET `/revisions/{revision_id}/documents`

GET `/revisions/{revision_id}/rendered-documents`

GET `/revisions`

GET `/revisions/{{revision_id}}`

POST `/revisions/{{revision_id}}/validations/{{name}}`

GET `/revisions/{{revision_id}}/validations`

GET `/revisions/{{revision_id}}/validations/{{name}}`

GET `/revisions/{{revision_id}}/validations/{{name}}/entries/{{entry_id}}`

DELETE `/docuemnts/{{schema}}/{{name}}`