This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.
Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
This PS simply reorganizes Deckhand's functional test directory
to make it more maintainable and readable as right now it is
hard to figure out what is covered by a functional test and
what isn't.
Additionally, the entrypoint for these tests in tools/functional-tests.sh
has also been refactored slightly.
Change-Id: I262c7e1f7cbce248c12ee013a9bab4e32b89adee
We will align the name with the rest of the UCP components, i.e.
change it from 'deckhand' to 'deckhand-api'
Change-Id: I4c65ac1e6371ffa80fd8b42cbe979d71b93e99c7
This patch set adds integration tests to Deckhand
where "integration" means the interaction between
Deckhand, Barbican and Keystone. OSH is used to
deploy Keystone and Barbican and Docker to deploy
PostgreSQL and Deckhand.
Unlike functional testing in Deckhand, all
integration tests use the default in-code policy
defaults and an admin token supplied by keystone
to validate authN and authZ.
The test scenarios consist of Deckhand secret
lifecycle management as well as document rendering
with secrets retrieved from Barbican.
Change-Id: Ib5ae1b345b2a4bd579671ec4ae9a232c2e3887dc
This patch set adds a relative symlink from docs/source/releasenotes
to releasenotes/source/ so that releasenotes can be published
to deckhand.readthedocs.io. A new tab link is added on the
documentation page which references Deckhand's release notes.
The command used to generate the symlink was:
ln -rs releasenotes/source/ docs/source/releasenotes
Change-Id: I07058ca47ecc7b1fb211cae53aedc5f796542cff
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.
Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.
Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
Add a blurb to the replacement documentation that differentiates
the layering replace action from document replacement to avoid
conflation of the terms.
Change-Id: Ie3da4645c5fa4ed0eab95184443acbb01c5b2740
Recently https://review.gerrithub.io/#/c/406626/ broke functional
tests via uwsgi because it changed how entrypoint.sh is called
which is performed during functional-tests.sh when uwsgi is used
to drive the server for testing. This changes how entrypoint.sh
is called so that the tests now pass with uwsgi.
Change-Id: I8252350676e61d5214da11e9ed282cc3399288d9
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.
Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.
Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
Follow up to [0] which includes document replacement
documentation. Includes documentation on the following
subjects:
* document replacement (new)
* document abstraction (new)
* document parent selection (edit)
This PS also renames some docs files by replacing "_"
with "-" in files with compound names for consistency.
[0] https://review.gerrithub.io/#/c/403888/
Change-Id: I6c1ba1e77347c266a6a9d471c9d7a747b1cef6eb
Under load, Deckhand will fail liveness checks with a 1 second timeout.
This Patchset extends the timout to 10 seconds and spaces the period
between checks to 20 seconds.
Adds labels to keystone user job.
Change-Id: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c
This PS integrates document replacement with document layering. The case
works something like this:
GIVEN:
- Parent A
- Child B
- Child C
WHEN:
- Child B is a replacement for A
THEN:
- B must layer with A, then C must layer with B, rather than A,
as B replaces A.
This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.
To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.
Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.
The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.
Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
This PS tests that Deckhand accepts and parses documents that use YAML
anchors and pointers. The restriction is that this be used intra-document
as anchors and pointers can't be used across documents.
Change-Id: I28e502a46e5fbb8cc27cf60e83d9b9e9572a3d36
This is a trivial PS that fixes the tox -e cover job in
tox.ini which was recently broken with [0].
[0] https://review.gerrithub.io/#/c/405318/
Change-Id: Id50a6348e6f306c3d8d68fdd79eb331880e7498b
Changes the entrypoint.sh options for uwsgi to include:
-b 32768 : for larger header/url handling
--die-on-term : for more 'normal' handling of SIGTERM
--lazy-apps : to delay init of python until after forking workers
--master : to provide a master process for handling request dispatch
The purpose of these changes is intended to avoid some crash behavior
that is occuring when the process being forked has an open db connection.
The --lazy-apps option should delay initialization. The other options are
recommended by uwsgi documentation, specicially the --master option.
The larger buffer size is not strictly recommended, but matters when large
headers are included.
The die-on-term option should provide better behavior in the container
environment.
Related-Change: I60adeffff5461fdda957124232bc5a606baae413
Change-Id: I70510246576a8fb6aa216e7c9c7e97c1c9ab791c
Layering code was not using a parse cache for jsonpath
This change adds use of the cache around all calls to jsonpath_ng.parse
Change-Id: I800eb397badf19ed2ea47b88fa7c91e4a09225ef
This sets multiple threads in Deckhand's chart config (4)
and set workers to just 1.
Deckhand's database is not configured to work with multiprocessing.
Currently there is a data race on acquiring shared SQLAlchemy
engine pooled connection strings when workers > 1. As a
workaround, we use multiple threads but only 1 worker. For more
information, see:
https://github.com/att-comdev/deckhand/issues/20
Change-Id: I60adeffff5461fdda957124232bc5a606baae413
This patch set updates the kubernetes-entrypoint image to version
3.0.3 inline of the chart used in OpenStack-Helm in [0]. This allows
the chart to use pod dependencies.
[0] https://review.openstack.org/#/c/554268/
Change-Id: I06c874bbe1b39271a94ce1c418c8b1317080dac5
Signed-off-by: Tin Lam <tin@irrational.io>
This is to add a helper function to base controller resource class
for detecting whether a list of documents has any empty (None)
or non-object entries in it (since all documents should be
objects), resulting in a 400 Bad Request getting raised.
This is to prevent the following stacktrace from occurring:
File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
responder(req, resp, **params)
File "./deckhand/policy.py", line 104, in handler
return func(*args, **kwargs)
File "./deckhand/control/buckets.py", line 58, in on_put
documents, data_schemas, pre_validate=True)
File "./deckhand/engine/document_validation.py", line 387, in __init__
raw_document[prop] = document.get(prop)
AttributeError: 'NoneType' object has no attribute 'get'
Change-Id: I76fc9b0d7662358f8b26b5bddf1187e92d1554de
This is to follow up on https://review.gerrithub.io/#/c/398708/
with more complete coverage for validation policy integration
with Validations API. The updated test covers both success and
failure paths.
Change-Id: I118c3783f98a3b38547745acc642d00d188f0b2e
This is to fix a minor synactical issue with one of the functional
tests causing the job to fail. The issue was introduced following
an improper rebase.
Change-Id: Ib3a3dd1baac572bdba3115cb4719f24d153db1e6
This is to log out all document data following any layering action
failure. This consists of two stages:
1) Scrubbing all primitives contained in the data sections of both
the child and parent being layered together.
2) Logging scrubbed-out data sections for both documents, in addition
to their names, schemas, and the layering action itself.
This will hopefully provide DEs with enough information about why
a layering action may have failed to apply while at the same time
preventing any secret data from being logged out.
Change-Id: I3fedd259bba7b930c7969e9c30d1fffef5bf77bd
This PS adds functional tests for the recent Validation Policy
integration with Validations API in
https://review.gerrithub.io/#/c/398708/
Change-Id: I6805d9a2bafaa6272038720aa215f13c25da3f49
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:
* generic case (a replaces b, returns a only)
* substitution case (a replaces b, c substitutes from a instead)
TODO:
* layering case (a replaces b, c layers with a instead)
* Modify Document unique constraint to work with
(schema, name, layer) throughout all of Deckhand
(including controllers, database models, and anywhere
else as needed)
Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.
The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
Deprecate substitution_sources from layering module because we
can just use the concrete documents as all the substitution
sources to simplify things.
Change-Id: Ibd8dff50402508417457655c367ebc9b6f28d70a
This adds a functional test for chained substitution.
For this test to pass a subtle bug had to be fixed in
the code in DocumentDict wrapper. It was possible for
instances of this class to return a NEW object when
`self.data` was invoked causing
`substitution_src.data.update(data)` to fail when
attempting to update a substitution source with updated
layered data because -- if data={} at that moment in time --
the code in DocumentDict would return a NEW object {}.
This was because `self.get('data') or {}` would return {}
on the RHS instead of self['data'] because self['data']
itself would be {} which would evaluate to false in terms
of truthiness resulting in the wrong object ref being returned...
essentially causing the update to fail. This has been
fixed.
Change-Id: I23ad0010e1d7df73e8e1a1456ba21b3e611bb0dd
UniqueConstraint is currently implemented incorrectly in terms
of syntax in Deckhand's Document DB model. This PS fixes that.
Now UniqueConstraint should be enforcing document uniqueness
at DB level such that an error is thrown for duplicate documents
(with same metadata.name and schema).
Closes #17
Change-Id: I7d66457f471ec48b5766733046977117b509d592
This is to skip layering for control documents (those whose
metadata.schema starts with "deckhand/Control") as these documents
consist of ValidationPolicy or LayeringPolicy documents -- and
it would be both nonsensical and scary to try to layer
these types of documents.
Documentation for this will be updated during a larger overhaul
effort to improve Deckhand's documentation.
Change-Id: Ia785e54c4e26a4158b6bdc89da8b96b4455f7b39
This is to update the ValidationPolicy documentation in Deckhand
to convey how it interacts with the Deckhand Validations API.
This is a follow up to: https://review.gerrithub.io/#/c/398708/
Change-Id: I45e043d56329f33d5ad0d65273ab282f03dc2d78
It's possible while running tests via testtools (python -m testtools.run)
that the file being imported in won't have registered all CONF opts
leading to oslo.conf errors getting thrown. This fixes that.
Change-Id: Ie21c89943bf4494ad9e0fd05e25f78139ad18377
Currently, if a document has a parent but no layering actions,
the document immediately inherents its parents' data, which is a
bug. Instead, the child document should only layer with its
parent's data and then update its own data if it has at least
one layering action.
In addition, the base_schema.yaml under `deckhand.schemas`
has been updated to require that actions be required and
contain at least 1 layering action when parentSelector
is provided and that parentSelector be required when
actions is provided and that at least one key-value
pair be provided. (Empty actions array or empty
parentSelector object is meaningless and should be
disallowed/discouraged.)
This means that actions and parentSelector must always
both be provided (though providing neither is also
legal because layering is optional).
Unit tests have been added to verify the schema updates.
Change-Id: I77d54e2b216efc54b466f94d82ee8d36ca169c26
This is to fix the condition in secrets_manager used to determine
whether the substitution is secret. It currently checks whether the
potential secret reference contains the substring
of 'key-manager/v1/secrets' but the environment-agnostic way
of doing this is to check whether the secret reference contains
the barbican endpoint registered under CONF.barbican.api_endpoint.
Change-Id: I633021571255c8393e19ec60a614ede981a86d9f
This fixes the Revision Resource class in the Deckhand client
incorrectly printing out revision documents as <Revision ID: {ID}>
instead of <Document [{schema}]: {name}>
Change-Id: I31024dda5803dbf35a23f9fb7bc224e0ca7d8ba2
This patch set does the following to enhance health/status checks
on the deckhand-api pod:
1) Add Liveness Probe
2) Update Readiness Probe
Change-Id: I1a1c107706862431e53668a864db622499e63c6f
This is to remove document data printout from the MissingDocumentKey
exception message which could expose sensitive data if it is caught
and logged by other services, for example. Instead, the child
and parent documents' schema and name are printed, in addition to
the action object in which the path that could not be resolved
in either parent or document is contained.
Change-Id: I07f43e57527d05e98e98e5f80567b97dd2a762f9
This PS integrates ValidationPolicy logic with the
Deckhand Validations API.
Support for multiple ValidationPolicy documents is
included.
If a ValidationPolicy is found, then the validations
contained therein are used to determine whether
a revision is successful or not. For example,
if a VP contains 'promenade-schema-validation' then
DH will return success if the externally registered
validation result for that validation is success.
However, if the result was never registered in DH
then the returned result is 'failure'.
In addition, if "extra" validations are registered
(that is validations not present in any VP) then
they are effectively ignored. An error message is
added with enough details to indicate why the validation
is ignored.
This PS adds unit tests to verify the correct behavior
for the above scenarios.
Functional tests and documentation changes will be added
in a follow up once design is ironed out.
Change-Id: I44c657974589ea3563e0a23ad667894329048b46
This is to add better logging to assist with debugging 500
Internal Server Errors that manifest from either internal
bugs in Deckhand or document typos/errors ingested by Deckhand
during document substitution.
Change-Id: I5f4c1cb07bea8e6546d08a858d4f83a24d75ef5a