This patch set adds `./entrypoint.sh alembic upgrade head` command
to manual install section in Getting Started guide, without which
Deckhand server cannot start correctly.
Change-Id: Id486cc98b8fd93271a8571364adfb9fb30bf4bcc
This patch set pins down package requirements for the following
reasons:
* Inconsistencies between ranges and direct pins
* Deckhand isn't gated against requirements repo so changes to
some packages isn't actively tested
* Other Airship components use pins, so Deckhand might break them
when package requirements go out of sync between components
* Deckhand as of yet has no stable release so Airship relies on
master for stability so it is better to ensure stability over
anything else
Change-Id: I20ef4be3e01cd267771098d33447ccb61aee20b4
This patch set reorganizes Deckhand's documentation structure
for better organization into 3 distinct categories:
* developer's guide
* operator's guide
* user's guide
This means that the RTD navigation menu on the left-hand side
will have fewer links (see list above) making navigation much
easier. This is similar to how Armada organizes its documentation
too.
This patch set also updates README section with a better
overview and trims some fat from it (remove testing
documentation as it doesn't really belong there -- there
is a dedicated page for that already).
Finally, this patch set changes the exceptions page to
render as a basic list of autoexception classes because
the current tabularized view is not rendering correctly
on RTD [0].
[0] https://airship-deckhand.readthedocs.io/en/latest/exceptions.html
Change-Id: I162383bf8e3bbd5004603c979ac7b0d760a210c4
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I12b2fa8fbec37a483a0ad50382e08f51ed97533a
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}
- Added deepdiff api for generating diff between
two rendered documents.
- Deep diffing for data and metadata
- Refactor diff functions
- Client update
- Added unit testcases
- Added funtional testcases
- Doc update
Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
Patterns may now be replaced recursively. This can be
achieved by using specifying a ``pattern`` value and
``recurse`` (with a required ``depth`` argument).
Example:
substitutions:
- dest:
path: .
pattern: REGEX
recurse:
depth: -1
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
NOTE:
Recursive selection of patterns will only consider
matching patterns. Non-matching patterns will be ignored.
Thus, even if recursion can "pass over" non-matching patterns,
they will be silently ignored.
This is useful for reducing the number of substitution
stanzas that are required for the purposes of performing
all the string pattern replacements that are required.
Best practice is to limit the scope of the recursion
as much as possible: e.g. avoid passing in "$" as the
``jsonpath``, but rather a JSON path that lives closer
to the nested strings in question.
Included in this patch set:
* recursive implementation for pattern replacement
* unit tests for most scenarios and edge cases
* docstring updates
* documentation updates
* schema updates for validation
Change-Id: I85048349097ed696667fae80f1180808d264bbcf
This patch set updates document types documentation
which includes expounding on control documents, layering policy
documents, and dataschema documents.
Change-Id: Id31fcb6b68ca30fdf681dec8348c2fc4237cd48d
This renames some instances of ucp to airship in the documentation
and makes some trivial documentation fixes.
Change-Id: I9a4a81d15bfc13b4fe089b7d65f0df43eeade9fb
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.
Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.
`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.
Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
- Treat internal Deckhand schemas equivalent to other
service schemas
- Remove validating sections other than `data` outside of
base schema
- Create schemas for metadata sections metadata/Control/v1 and
metadata/Document/v1
- Use a single validator and let that validator check for document
structure (validate against the base schema and metadata)
and for post-validation also validate against service schemas
Change-Id: I5f9b9a3cfa1692a69b5982a6424edd65bdfed0ef
This patchset adds developer overview documentation for providing
a high-level introduction to Deckhand, including its architecture,
modules, test utilities, Helm utilities, and other errata. This
work is based off this Drydock patchset:
https://review.openstack.org/#/c/571298/
Change-Id: Ic3382d4e04edf02a65184651d272fe9cd1db56a4
This patchset expands on document uniquness in Deckhand which is
a bit involved.
At the **database** level, documents are uniquely identified by
the combination of:
#. ``metadata.name``
#. ``schema``
#. ``metadata.layeringDefinition.layer``
This means that raw revision documents -- which are persisted
in Deckhand's database -- require that the combination of all
3 parameters be unique.
However, **post-rendered documents** are only uniquely
identified by the combination of:
#. ``metadata.name``
#. ``schema``
Because collisions with respect to the the third parameter --
``metadata.layeringDefinition.layer`` -- can only occur with
replacement. But after document rendering, the replacement-parent
documents are never returned.
Change-Id: I841caa83bef46bf5e155136fbcd7444b672d06f8
This PS adds an integration test scenario for validating that
encrypting a generic document type and using it as a substitution
source during document rendering works.
Deckhand will now submit all generic documents to be encrypted
to Barbican with a 'secret_type' of 'passphrase'. No encoding
is provided Deckhand-side (i.e. base64) because encoding is
deprecated in Barbican since it lead to strange behavior;
Barbican will figure out what to encode the payload as
automatically. For more information, see [0] and [1].
In addition, this PS handles 2 edge cases around secret
payloads that are rejected by Barbican if not handled
correctly by Deckhand: empty payloads and non-string
type payloads [2]. For the first case Deckhand forcibly
changes the document to cleartext because there is no
point in encrypting a document with an empty payload.
For the second case Deckhand sets overrides any
previously set secret_type to 'opaque' and encodes
the payload to base64 -- when it goes to render
the secret it decodes the payload also using base64.
Integration tests have been added to handle both edge
cases described above.
[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1419166
[1] 49505b9aec/barbicanclient/v1/secrets.py (L252)
[2] 49505b9aec/barbicanclient/v1/secrets.py (L297)
Change-Id: I1964aa84ad07b6f310b39974f078b84a1dc84983
This patchset adds documentation on document encryption. It also
adds appropriate references to other sections of the documentation
where linking to document encryption is apropos. Finally, this
patchset adds further information and structure around the document
rendering process in general, in order to underscore the pieces
of documentation that undergird the general concept of document
rendering: substitution, layering and replacement.
Change-Id: I566cac4a3374556a0e62e0e1962e153029f7ec05
This patchset updates docs to doc to align with OpenStack
standard. Follow-up patchset will be needed to publish
documentation to OpenStack [0].
[0] https://docs.openstack.org/doc-contrib-guide/project-guides.html
Change-Id: Ia191ac1cc4536af1232aedd4bb491f3829651730
This PS integrates ValidationPolicy logic with the
Deckhand Validations API.
Support for multiple ValidationPolicy documents is
included.
If a ValidationPolicy is found, then the validations
contained therein are used to determine whether
a revision is successful or not. For example,
if a VP contains 'promenade-schema-validation' then
DH will return success if the externally registered
validation result for that validation is success.
However, if the result was never registered in DH
then the returned result is 'failure'.
In addition, if "extra" validations are registered
(that is validations not present in any VP) then
they are effectively ignored. An error message is
added with enough details to indicate why the validation
is ignored.
This PS adds unit tests to verify the correct behavior
for the above scenarios.
Functional tests and documentation changes will be added
in a follow up once design is ironed out.
Change-Id: I44c657974589ea3563e0a23ad667894329048b46
This PS removes microversions from document versions because
no services use microversions and microversioning for documents
isn't supported in Deckhand.
Change-Id: I3635d15513a2c7b8154ec6be4d0b8577e7d4ce3d
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.
The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.
If pre_validate is true, then:
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* NO externally registered DataSchema documents
are used for validation
Else (if pre_validate is false):
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* ALL externally registered DataSchema documents
are used for validation given a schema match
A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.
Change-Id: I859f61989ec15bede1c104b86625d116064f056d
This is to fix up a slight rendering issue with getting
started documentation related to double code block syntax.
Change-Id: I4b6777f77d148113e5a798aeb3daa520d05c1f0e
Update the README by moving a lot of the manual installation into
Getting Started docs and simplify some of the documentation because
Deckhand can be re-run using sqlite (for dev purposes, for example).
Change-Id: I5742dc75b95e2af67a18b419d04e769c10a1e43e
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)
Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.
Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.
Unit tests were added and documentation changes were made.
Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
This ps adds the full path for Sphinx so the automodule classes
will show properly on Readthedocs.
Change-Id: I337e5b9a90d3a9d99f4e8270271fcf7ddc67b305
This patch set fixes a handful of typographical errors in the documents.
Change-Id: Ic2b590dcf4d16ae35ed88bdc606d4bcfb2d9dd58
Signed-off-by: Tin Lam <tin@irrational.io>
This PS moves DB calls out of the engine module and into the
appropriate controllers so that a "production" set of documents
can leverage Deckhand layering and substitution after importing
the Deckhand engine module directly. These operations will be
carried out offline meaning that DB calls are not possible.
Unit tests were refactored to work with the changes.
Some testing documentation was also updated.
Closes 16
Change-Id: I6e0757746cd949985d57102d1c85acfbbed86078
This PS implements sorting and filtering for rendered-documents
endpoint, adds additional validations for sorting, filtering
and other layering scenarios, and updates rendered-documents
and buckets documentation.
Layering scenarios added:
- Updating the LayeringPolicy with 2 layers in the layerOrder
(down from 3) such that the site document should have its
parent document recomputed as the global document.
- A deletion action layering scenario (DH currently only has
merge, replace scenarios in its funcitonal test suite.)
Documentation updated:
- clarify the access levels for buckets, which has been a
source of confusion.
- update api-ref documentation for rendered-documents
Change-Id: Idb9b42351dfbdf75a19282c8478065e7564cfc26
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.
This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.
This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.
Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.
Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
This PS adds functional tests for built-in or "owned"
Deckhand documents. This includes 4 new document types
for which schemas were created:
- CertificateAuthority
- CertificateAuthority
- PublicKey
- PrivateKey
These new types are required by Promenade.
Finally, a bug was fixed which was causing secrets to
return as {'secret': "original secret payload"} which has
been fixed to return as simply "original secret payload".
Change-Id: Ifb4d41f5f4ac96b3103210853ad763d766ace93e
This PS simply adds a documentation blurb about how to use the
Decklahdn client via a pre-existing Keystone token.
Other auth plugins can be used with a similar effect. See:
edeb84715c/keystoneauth1/identity/v3
Change-Id: I5bf5a57ec02a7b2bda7304b5c5068f88cea228e1
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
This PS implements the Deckhand API client library
which is based off the python-novaclient code base.
The client library includes managers for all the
Deckhand APIs.
The following features have been implemented:
* Framework for API client library
* Manager for each Deckhand API (buckets, revisions, etc.)
* API client library documentation
Tests will be added in a follow-up (once Deckhand functional
tests use Keystone).
Change-Id: I829a030738f42dc7ddec623d881a99ed97d04520
