375 Commits

Author SHA1 Message Date
Felipe Monteiro
5c078ce81c Update releasenotes/docs tox jobs
This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.

Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
2018-04-24 22:34:49 +01:00
Felipe Monteiro
1566b9541a Clean up functional test directory and entrypoint script
This PS simply reorganizes Deckhand's functional test directory
to make it more maintainable and readable as right now it is
hard to figure out what is covered by a functional test and
what isn't.

Additionally, the entrypoint for these tests in tools/functional-tests.sh
has also been refactored slightly.

Change-Id: I262c7e1f7cbce248c12ee013a9bab4e32b89adee
2018-04-20 22:07:04 +01:00
Anthony Lin
c29ad4406b Change name of Deckhand Container
We will align the name with the rest of the UCP components, i.e.
change it from 'deckhand' to 'deckhand-api'

Change-Id: I4c65ac1e6371ffa80fd8b42cbe979d71b93e99c7
2018-04-19 23:25:20 -04:00
Felipe Monteiro
f30484a14c Add integration tests
This patch set adds integration tests to Deckhand
where "integration" means the interaction between
Deckhand, Barbican and Keystone. OSH is used to
deploy Keystone and Barbican and Docker to deploy
PostgreSQL and Deckhand.

Unlike functional testing in Deckhand, all
integration tests use the default in-code policy
defaults and an admin token supplied by keystone
to validate authN and authZ.

The test scenarios consist of Deckhand secret
lifecycle management as well as document rendering
with secrets retrieved from Barbican.

Change-Id: Ib5ae1b345b2a4bd579671ec4ae9a232c2e3887dc
2018-04-18 09:05:04 -04:00
Felipe Monteiro
d32c7a2c8d [docs] Publish releasenotes alongside docs to readthedocs
This patch set adds a relative symlink from docs/source/releasenotes
to releasenotes/source/ so that releasenotes can be published
to deckhand.readthedocs.io. A new tab link is added on the
documentation page which references Deckhand's release notes.

The command used to generate the symlink was:

    ln -rs releasenotes/source/ docs/source/releasenotes

Change-Id: I07058ca47ecc7b1fb211cae53aedc5f796542cff
2018-04-15 14:09:06 -04:00
Felipe Monteiro
106038d3cd [fix] Pass secret URI instead of UUID to barbican get_secret
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.

While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].

API ref:

GET /v1/secrets/{uuid}
Headers:
    Accept: application/json
    X-Auth-Token: {token}
    (or X-Project-Id: {project_id})

CLI ref:

$  barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
                           [--max-width <integer>] [--prefix PREFIX]
                           [--decrypt] [--payload]
                           [--payload_content_type PAYLOAD_CONTENT_TYPE]
                           URI

Retrieve a secret by providing its URI.

Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.

[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get

Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
2018-04-11 17:33:58 -04:00
Mark Burnett
7086973386 Add negative functional test for substitution
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.

Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.

Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
2018-04-11 14:04:22 -04:00
Felipe Monteiro
6b9cd17e98 docs: Distinguish replace layering action from document replacement
Add a blurb to the replacement documentation that differentiates
the layering replace action from document replacement to avoid
conflation of the terms.

Change-Id: Ie3da4645c5fa4ed0eab95184443acbb01c5b2740
2018-04-10 21:52:00 -04:00
Felipe Monteiro
e23f46c152 Fix running functional tests via uwsgi
Recently https://review.gerrithub.io/#/c/406626/ broke functional
tests via uwsgi because it changed how entrypoint.sh is called
which is performed during functional-tests.sh when uwsgi is used
to drive the server for testing. This changes how entrypoint.sh
is called so that the tests now pass with uwsgi.

Change-Id: I8252350676e61d5214da11e9ed282cc3399288d9
2018-04-10 16:48:03 -04:00
Felipe Monteiro
a3d32c3459 Raise exception on unfound secret in source document
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.

Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
2018-04-09 17:04:58 +01:00
Felipe Monteiro
35f15ee601 [fix] Drop deckhand.conf from default DECKHAND_CONF_DIR path
This drops deckhand.conf from default DECKHAND_CONF_DIR path so
that:

exec uwsgi \
    ...
    --pyargv "--config-file ${DECKHAND_CONFIG_DIR}/deckhand.conf" \
    ...

Doesn't append deckhand.conf again, resulting in a broken path like:
/etc/deckhand/deckhand.conf/deckhand.conf

Change-Id: I1c9d5ad283fed7158ba70b84b5b4afb27a3ab5d9
2018-04-07 00:20:47 -04:00
Bryan Strassner
5f1fbbee3c [396582] Add alembic support to Deckhand
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.

Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
2018-04-06 23:30:16 -04:00
Scott Hussey
4d90257372 [Fix] Multidigit array index
- Allow array index in substitution destinations to have
  multiple digits

Change-Id: I8ef6241763dd7d841e25774fa041f7f4a3b11c7b
2018-04-05 16:56:30 -05:00
Felipe Monteiro
24b4cf0420 Document replacement documentation
Follow up to [0] which includes document replacement
documentation. Includes documentation on the following
subjects:

  * document replacement (new)
  * document abstraction (new)
  * document parent selection (edit)

This PS also renames some docs files by replacing "_"
with "-" in files with compound names for consistency.

[0] https://review.gerrithub.io/#/c/403888/

Change-Id: I6c1ba1e77347c266a6a9d471c9d7a747b1cef6eb
2018-04-05 15:09:51 -04:00
Bryan Strassner
c962eeb975 [fix] Extend liveness and readiness check times
Under load, Deckhand will fail liveness checks with a 1 second timeout.
This Patchset extends the timout to 10 seconds and spaces the period
between checks to 20 seconds.
Adds labels to keystone user job.

Change-Id: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c
2018-04-04 15:31:46 -05:00
Felipe Monteiro
74528a518d Document replacement: Layering dependency integration
This PS integrates document replacement with document layering. The case
works something like this:

  GIVEN:
  - Parent A
  - Child B
  - Child C

  WHEN:
  - Child B is a replacement for A

  THEN:
  - B must layer with A, then C must layer with B, rather than A,
    as B replaces A.

This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.

To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.

Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.

The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.

Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
2018-04-04 10:58:28 -04:00
Felipe Monteiro
45441b6618 Test that Deckhand works with YAML anchors/pointers
This PS tests that Deckhand accepts and parses documents that use YAML
anchors and pointers. The restriction is that this be used intra-document
as anchors and pointers can't be used across documents.

Change-Id: I28e502a46e5fbb8cc27cf60e83d9b9e9572a3d36
2018-04-03 16:22:58 -04:00
Felipe Monteiro
4a586e370e Remove unused functions from DB module
This is a trivial PS that removes 2 unused functions from the
DB module.

Change-Id: I5c84d0b9d7b4c7e64a3608dcc5cc5262e0b9424a
2018-04-03 14:42:48 -04:00
Felipe Monteiro
6a521e2eb9 Trivial fix: Fix coverage tox.ini job
This is a trivial PS that fixes the tox -e cover job in
tox.ini which was recently broken with [0].

[0] https://review.gerrithub.io/#/c/405318/

Change-Id: Id50a6348e6f306c3d8d68fdd79eb331880e7498b
2018-04-03 14:17:58 -04:00
Bryan Strassner
4d642f849a [fix] Add uwsgi entrypoint options
Changes the entrypoint.sh options for uwsgi to include:
 -b 32768 : for larger header/url handling
 --die-on-term : for more 'normal'  handling of SIGTERM
 --lazy-apps : to delay init of python until after forking workers
 --master : to provide a master process for handling request dispatch

The purpose of these changes is intended to avoid some crash behavior
that is occuring when the process being forked has an open db connection.
The --lazy-apps option should delay initialization. The other options are
recommended by uwsgi documentation, specicially the --master option.
The larger buffer size is not strictly recommended, but matters when large
headers are included.

The die-on-term option should provide better behavior in the container
environment.

Related-Change: I60adeffff5461fdda957124232bc5a606baae413
Change-Id: I70510246576a8fb6aa216e7c9c7e97c1c9ab791c
2018-04-03 09:19:28 -04:00
Bryan Strassner
c50501cc89 [fix] Updates to use cached jsonpath
Layering code was not using a parse cache for jsonpath
This change adds use of the cache around all calls to jsonpath_ng.parse

Change-Id: I800eb397badf19ed2ea47b88fa7c91e4a09225ef
2018-04-02 16:30:06 -05:00
Felipe Monteiro
5c9efa9d74 Enable multiple threads, disabled muliple workers
This sets multiple threads in Deckhand's chart config (4)
and set workers to just 1.

Deckhand's database is not configured to work with multiprocessing.
Currently there is a data race on acquiring shared SQLAlchemy
engine pooled connection strings when workers > 1. As a
workaround, we use multiple threads but only 1 worker. For more
information, see:

https://github.com/att-comdev/deckhand/issues/20

Change-Id: I60adeffff5461fdda957124232bc5a606baae413
2018-04-02 12:38:20 -04:00
Tin Lam
225638711b Update kubernetes-entrypoint
This patch set updates the kubernetes-entrypoint image to version
3.0.3 inline of the chart used in OpenStack-Helm in [0]. This allows
the chart to use pod dependencies.

[0] https://review.openstack.org/#/c/554268/

Change-Id: I06c874bbe1b39271a94ce1c418c8b1317080dac5
Signed-off-by: Tin Lam <tin@irrational.io>
2018-04-02 10:22:30 -04:00
Felipe Monteiro
04ad3fa93b Add validation for empty documents inside multi-document payload
This is to add a helper function to base controller resource class
for detecting whether a list of documents has any empty (None)
or non-object entries in it (since all documents should be
objects), resulting in a 400 Bad Request getting raised.
This is to prevent the following stacktrace from occurring:

File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
responder(req, resp, **params)
File "./deckhand/policy.py", line 104, in handler
return func(*args, **kwargs)
File "./deckhand/control/buckets.py", line 58, in on_put
documents, data_schemas, pre_validate=True)
File "./deckhand/engine/document_validation.py", line 387, in __init__
raw_document[prop] = document.get(prop)
AttributeError: 'NoneType' object has no attribute 'get'

Change-Id: I76fc9b0d7662358f8b26b5bddf1187e92d1554de
2018-03-31 22:36:55 -04:00
Felipe Monteiro
99e3064eda [test] Improve validation policy test coverage for success scenario
This is to follow up on https://review.gerrithub.io/#/c/398708/
with more complete coverage for validation policy integration
with Validations API. The updated test covers both success and
failure paths.

Change-Id: I118c3783f98a3b38547745acc642d00d188f0b2e
2018-03-30 10:21:09 -04:00
Anthony Lin
28254bc7ff Update Makefile - Dryrun
Add template for deckhand chart dry-run

Change-Id: I9eab723a9ad905627c993aafdbf12b62c7dfc865
2018-03-29 21:22:23 -04:00
Felipe Monteiro
32ad739e68 [TrivialFix] Unblock gate due to failing test after rebase
This is to fix a minor synactical issue with one of the functional
tests causing the job to fail. The issue was introduced following
an improper rebase.

Change-Id: Ib3a3dd1baac572bdba3115cb4719f24d153db1e6
2018-03-29 13:38:26 -04:00
Felipe Monteiro
a5f75722dc Log all document data following any layering action failure
This is to log out all document data following any layering action
failure. This consists of two stages:

1) Scrubbing all primitives contained in the data sections of both
   the child and parent being layered together.
2) Logging scrubbed-out data sections for both documents, in addition
   to their names, schemas, and the layering action itself.

This will hopefully provide DEs with enough information about why
a layering action may have failed to apply while at the same time
preventing any secret data from being logged out.

Change-Id: I3fedd259bba7b930c7969e9c30d1fffef5bf77bd
2018-03-29 10:10:03 -04:00
Felipe Monteiro
2b5848a273 Add functional tests for Validation Policy changes
This PS adds functional tests for the recent Validation Policy
integration with Validations API in
https://review.gerrithub.io/#/c/398708/

Change-Id: I6805d9a2bafaa6272038720aa215f13c25da3f49
2018-03-29 10:02:48 -04:00
Felipe Monteiro
84435cd1c8 Add functional tests for document replacement
Covers replacement of a substitution source working.

Change-Id: I7c3fc1b4f8477df2bb932f43c6033eaf9c6ceb39
2018-03-28 17:09:40 -04:00
Felipe Monteiro
4799acdbcc Engine implementation for document replacement
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:

  * generic case (a replaces b, returns a only)
  * substitution case (a replaces b, c substitutes from a instead)

TODO:

  * layering case (a replaces b, c layers with a instead)
  * Modify Document unique constraint to work with
    (schema, name, layer) throughout all of Deckhand
    (including controllers, database models, and anywhere
     else as needed)

Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
2018-03-28 17:09:09 -04:00
Felipe Monteiro
1264e5af6c Document replacement: Update Document unique constraint
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.

The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/

Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
2018-03-28 17:08:03 -04:00
Felipe Monteiro
9cae8f3b2b Switch to stestr
stestr is actively maintained. OpenStack is migrating to stestr,
so we should follow suit.

Change-Id: I1b958681cbf53281c9da91c747c9a2cfe2beccf3
2018-03-28 13:06:46 -04:00
Anthony Lin
f179167e9a [398395] Update Indentation for Resource limits
Update indentation for 'test-deckhand-api'

Change-Id: I0c2ba99c6743f5e7b7cdf26e74479a4ebfe744a1
2018-03-28 13:45:36 +00:00
Felipe Monteiro
91de02be34 Fix secret_uuid used to query Barbican's Secrets API
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.

The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.

[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid

Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
2018-03-26 21:30:57 +01:00
Felipe Monteiro
d86d87d16c Deprecate substitution_sources from layering module
Deprecate substitution_sources from layering module because we
can just use the concrete documents as all the substitution
sources to simplify things.

Change-Id: Ibd8dff50402508417457655c367ebc9b6f28d70a
2018-03-26 21:26:32 +01:00
Mark Burnett
44114dad3b Add functional test for chained substitution
This adds a functional test for chained substitution.

For this test to pass a subtle bug had to be fixed in
the code in DocumentDict wrapper. It was possible for
instances of this class to return a NEW object when
`self.data` was invoked causing
`substitution_src.data.update(data)` to fail when
attempting to update a substitution source with updated
layered data because -- if data={} at that moment in time --
the code in DocumentDict would return a NEW object {}.

This was because `self.get('data') or {}` would return {}
on the RHS instead of self['data'] because self['data']
itself would be {} which would evaluate to false in terms
of truthiness resulting in the wrong object ref being returned...
essentially causing the update to fail. This has been
fixed.

Change-Id: I23ad0010e1d7df73e8e1a1456ba21b3e611bb0dd
2018-03-26 14:42:50 -04:00
Felipe Monteiro
cce6ddaf6e Fix uniqueness not being enforced at DB level for documents
UniqueConstraint is currently implemented incorrectly in terms
of syntax in Deckhand's Document DB model. This PS fixes that.
Now UniqueConstraint should be enforcing document uniqueness
at DB level such that an error is thrown for duplicate documents
(with same metadata.name and schema).

Closes #17

Change-Id: I7d66457f471ec48b5766733046977117b509d592
2018-03-22 05:57:19 -04:00
Felipe Monteiro
d20f4741c5 Skip layering for control documents
This is to skip layering for control documents (those whose
metadata.schema starts with "deckhand/Control") as these documents
consist of ValidationPolicy or LayeringPolicy documents -- and
it would be both nonsensical and scary to try to layer
these types of documents.

Documentation for this will be updated during a larger overhaul
effort to improve Deckhand's documentation.

Change-Id: Ia785e54c4e26a4158b6bdc89da8b96b4455f7b39
2018-03-21 15:47:22 -05:00
Felipe Monteiro
a691f9a47f Add readthedocs link to Deckhand readme
Change-Id: Id3aeccec0730fe0556446d42facfd222751d636c
2018-03-16 12:41:20 -04:00
Felipe Monteiro
14d4dee09f Docs: Update ValidationPolicy documentation
This is to update the ValidationPolicy documentation in Deckhand
to convey how it interacts with the Deckhand Validations API.
This is a follow up to: https://review.gerrithub.io/#/c/398708/

Change-Id: I45e043d56329f33d5ad0d65273ab282f03dc2d78
2018-03-16 11:20:18 -04:00
Felipe Monteiro
9a5198fef5 Trivial: Add import to base unit test to register CONF opts
It's possible while running tests via testtools (python -m testtools.run)
that the file being imported in won't have registered all CONF opts
leading to oslo.conf errors getting thrown. This fixes that.

Change-Id: Ie21c89943bf4494ad9e0fd05e25f78139ad18377
2018-03-16 02:45:06 +00:00
Felipe Monteiro
5c411dd05b Fix: Document should not layer with parent if no layering actions
Currently, if a document has a parent but no layering actions,
the document immediately inherents its parents' data, which is a
bug. Instead, the child document should only layer with its
parent's data and then update its own data if it has at least
one layering action.

In addition, the base_schema.yaml under `deckhand.schemas`
has been updated to require that actions be required and
contain at least 1 layering action when parentSelector
is provided and that parentSelector be required when
actions is provided and that at least one key-value
pair be provided. (Empty actions array or empty
parentSelector object is meaningless and should be
disallowed/discouraged.)

This means that actions and parentSelector must always
both be provided (though providing neither is also
legal because layering is optional).

Unit tests have been added to verify the schema updates.

Change-Id: I77d54e2b216efc54b466f94d82ee8d36ca169c26
2018-03-15 12:27:31 -04:00
Felipe Monteiro
d27814cb1e Trivial: Rename doc to docs to align with UCP standard
Change-Id: I79e1544bfee8701cc49fe50509d69d53495e2ffa
2018-03-15 15:03:29 +00:00
Felipe Monteiro
116fafcec3 Fix condition for checking whether substitution is secret
This is to fix the condition in secrets_manager used to determine
whether the substitution is secret. It currently checks whether the
potential secret reference contains the substring
of 'key-manager/v1/secrets' but the environment-agnostic way
of doing this is to check whether the secret reference contains
the barbican endpoint registered under CONF.barbican.api_endpoint.

Change-Id: I633021571255c8393e19ec60a614ede981a86d9f
2018-03-14 21:51:28 +00:00
Felipe Monteiro
bf70a81ffa Fix Revision Resource print out in Deckhand client
This fixes the Revision Resource class in the Deckhand client
incorrectly printing out revision documents as <Revision ID: {ID}>
instead of <Document [{schema}]: {name}>

Change-Id: I31024dda5803dbf35a23f9fb7bc224e0ca7d8ba2
2018-03-13 22:31:38 -04:00
Anthony Lin
a29741e3a7 Deckhand API - Liveness and Readiness Probes
This patch set does the following to enhance health/status checks
on the deckhand-api pod:

1) Add Liveness Probe
2) Update Readiness Probe

Change-Id: I1a1c107706862431e53668a864db622499e63c6f
2018-03-13 15:31:52 +00:00
Felipe Monteiro
e90c0aedf8 Security fix: Remove document data printout from exception message
This is to remove document data printout from the MissingDocumentKey
exception message which could expose sensitive data if it is caught
and logged by other services, for example. Instead, the child
and parent documents' schema and name are printed, in addition to
the action object in which the path that could not be resolved
in either parent or document is contained.

Change-Id: I07f43e57527d05e98e98e5f80567b97dd2a762f9
2018-03-12 20:42:31 +00:00
Felipe Monteiro
d82d0cfaf7 ValidationPolicy integration with Validations API
This PS integrates ValidationPolicy logic with the
Deckhand Validations API.

Support for multiple ValidationPolicy documents is
included.

If a ValidationPolicy is found, then the validations
contained therein are used to determine whether
a revision is successful or not. For example,
if a VP contains 'promenade-schema-validation' then
DH will return success if the externally registered
validation result for that validation is success.
However, if the result was never registered in DH
then the returned result is 'failure'.

In addition, if "extra" validations are registered
(that is validations not present in any VP) then
they are effectively ignored. An error message is
added with enough details to indicate why the validation
is ignored.

This PS adds unit tests to verify the correct behavior
for the above scenarios.

Functional tests and documentation changes will be added
in a follow up once design is ironed out.

Change-Id: I44c657974589ea3563e0a23ad667894329048b46
2018-03-12 12:41:06 -04:00
Felipe Monteiro
65c459d1f9 Improve secrets_manager logging after 500 Internal Server Error
This is to add better logging to assist with debugging 500
Internal Server Errors that manifest from either internal
bugs in Deckhand or document typos/errors ingested by Deckhand
during document substitution.

Change-Id: I5f4c1cb07bea8e6546d08a858d4f83a24d75ef5a
2018-03-10 16:39:48 -05:00