This adds a uwsgi functional test check to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.
Change-Id: I931ab4d11719daca7665d3a25b00e353c707237e
This patchset converts much of the previous logic in
functional-tests.sh into Ansible playbooks to be executed
by Zuul. This mainly includes all the Docker-related
deployment logic.
The functional-tests.sh script has been slimmed down to
just work with uwsgi so that a standalone functional
test deployment can be performed relatively easily,
mainly by developers.
Finally, py27 support for the gate has been dropped
as the Dockerfile in this project currently assumes
python3 for installing requirements and so forth,
leading to requirements issues blocking the gate.
Change-Id: I903a2845390061641d292fb0c016ba6a53723fc9
Prometheus scrapers aren't running in the airship-deckhand
deployment job, so no need to include this here. It is
mainly being removed because of a POST_FAILURE issue
resulting from including the role in question in this job.
Change-Id: Ia080ff333c82f5b4b13d2e5db8c88741c0dc4613
This patchset adds functional tests to .zuul.yaml. Additionally
it adds a functional-py35 job as well which will also be kicked
off via Zuul.
Change-Id: Ic2d1db4d3cd65c4d93c3a6f04e6efeeba9755f07
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.
Change-Id: If5b6550c6f0d9d1f96c4822e72d7d519dfca3c79
Signed-off-by: Pete Birley <pete@port.direct>
Currently it doesn't seem document replacement works
exactly as expected: The parent-replacement document
can receive layering and substitution data prior to
being replaced. Currently, Deckhand does not account
for this scenario.
A child-replacement depends on its parent-replacement
the same way any child depends on its parent: so that the
child layers with its parent only after the parent has
received all layering and substitution data. But other
documents that depend on the parent-replacement actually
depend on the child-replacement instead as the
child-replacement replaces its parent. So the dependency
chain is: PR -> CR -> anything that layers with PR.
A unit and functional test have been added for regression.
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I353393f416aa6e441d84add9ebedcd152944d7e8
As part of ongoing effort to update the "application" and
"component" labels for the UCP components, there is a need
to align with the convention. We will update the label for
the deckhand API pod in this case.
Also updated helm_tk.sh to point to openstack-helm-infra for
reference to helm-toolkit as helm-toolkit has been removed
from the openstack-helm repo [0]
[0] https://review.openstack.org/#/c/558065/
Change-Id: I753c4ce653790250b79986c670224d0962f7676f
This patchset updates the Apache LICENSE to be consistent with
other OpenStack services [0]. This in effect drops the appendix
section not present in the OpenStack license.
https://github.com/openstack/nova/blob/master/LICENSE
Change-Id: I9b350c73d92633f45cbd9c32f9b8740fde3946df
This is to stop the DH pod from being killed in production whenever
DH receives multiple concurrent requests from another service,
causing all its threads to become occupied with servicing those
requests, causing the liveness probe to fail, causing the DH pod
to be killed. This is highly undesirable and as a temporary
workaround we will drop the liveness probe altogether.
This partially reverts I1a1c107706862431e53668a864db622499e63c6f
Additional reading: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c
Change-Id: Ic81c0c1d6e3cd3ab3b326054b9c882962d240968
This PS adds a limit query filter parameter to allow users to limit
the number of documents returned by revision documents as well
as rendered documents.
Change-Id: Ic15dc59cd21d82be552fa7b9885754bde47724a0
Fix failing pep8 errors which were never being flagged but now are,
possibly due to changes in flake8 rules. This patchset corrects
the following errors:
./deckhand/engine/layering.py:567:21: W503 line break before binary operator
./deckhand/engine/secrets_manager.py:406:33: W503 line break before binary operator
./deckhand/engine/utils.py:33:17: W503 line break before binary operator
./deckhand/common/utils.py:292:17: W503 line break before binary operator
Change-Id: Ic26aecb6b8049e138a826af9953f45298e817795
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.
The logic is similar to Drydock's here: [0].
[0] 1c78477e95/drydock_provisioner/util.py (L43)
Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
Due to a rebase issue, integration tests are skipping which is
unintended and undesirable. This fixes the issue.
Change-Id: I8e37f4610e4a1e9b8fb4fd81aa8643100cd6dfd5
This PS:
* adds a trap to clean up OSH which is deployed in the
course of integration tests. It appears as though node cleanup
in Jenkins is hanging so this is to try to ameliorate that
* creates a deckhand.conf.test to be used by functional and
integration tests instead of writing it out dynamically [0]
* updates logging.conf.sample to dump logs to stdout/stderr
by default as this is amenable to containers
* makes test_gabbi.py common between functional and integration
tests to avoid unnecessary code duplication
[0] review comments in https://review.gerrithub.io/#/c/att-comdev/deckhand/+/407638/
Change-Id: I762fb0bde5f75effcde56316d92bd57b30026995
This PS simply updates the README to correct typos (intgration =>
integration) and reword misleading sections, as well as update
deprecated sections. Finally, new sections related to Barbican
have been added.
Change-Id: I92611cf2aecf5ee1295ba9014002cd0e18f3a6af
This patch set adds a new endpoint to the Validations API which allows
for listing all validations for a given revision with details.
The response body for GET /api/v1.0/{revision_id}/validations/detail
looks like:
---
count: 1
next: null
prev: null
results:
- name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
status: failure
createdAt: 2017-07-16T02:03Z
expiresAfter: null
expiresAt: null
errors:
- documents:
- schema: promenade/Node/v1
name: node-document-name
- schema: promenade/Masters/v1
name: kubernetes-masters
message: Node has master role, but not included in cluster masters list.
Note that the Validations API in general is currently missing fields
like url (as well as next and prev references) which will be included
in a follow up.
This will enable Shipyard to avoid performing a quadratic number
of API look ups when querying Deckhand's Validations API: [0].
The policy enforced for this endpoint is deckhand:list_validations.
APIImpact
DocImpact
[0] 06b5e82ea8/shipyard_airflow/control/configdocs/deckhand_client.py (L265)
Change-Id: I827e5f47bffb23fa16ee5c8a705058034633baed
This patch set accomplishes 2 things:
1) Fixes an issue in Deckhand caused by improperly using the wrong
Barbican secret type for the PrivateKey Deckhand document type.
2) Tests all Deckhand secret types against Barbican via integration
testing.
The current error being raised is this: "Provided object does not match
schema 'Secret': u'privatekey' is not one of ['symmetric', 'passphrase',
'private', 'public', 'certificate', 'opaque']."
Change-Id: I8231c87782902850fe0632d0346c567c7481e95f
This is to fix a use case where a subpath like 'filter:authtoken' in
a JSON path like ".values.conf.paste.'filter:authtoken'.password" fails
because the resulting Python object that is created in memory
for substitution constructs a dict key like "'filter:authtoken'" resulting
in Deckhand failing to index into the dict as the key needs to be
stripped of start and end quotes that are only necessary for achieving
valid YAML syntax.
A unit test is added for regression.
Change-Id: I19974efc977b0cdc5793e649fa068d1a3bd7339e
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.
Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
This PS simply reorganizes Deckhand's functional test directory
to make it more maintainable and readable as right now it is
hard to figure out what is covered by a functional test and
what isn't.
Additionally, the entrypoint for these tests in tools/functional-tests.sh
has also been refactored slightly.
Change-Id: I262c7e1f7cbce248c12ee013a9bab4e32b89adee
We will align the name with the rest of the UCP components, i.e.
change it from 'deckhand' to 'deckhand-api'
Change-Id: I4c65ac1e6371ffa80fd8b42cbe979d71b93e99c7
This patch set adds integration tests to Deckhand
where "integration" means the interaction between
Deckhand, Barbican and Keystone. OSH is used to
deploy Keystone and Barbican and Docker to deploy
PostgreSQL and Deckhand.
Unlike functional testing in Deckhand, all
integration tests use the default in-code policy
defaults and an admin token supplied by keystone
to validate authN and authZ.
The test scenarios consist of Deckhand secret
lifecycle management as well as document rendering
with secrets retrieved from Barbican.
Change-Id: Ib5ae1b345b2a4bd579671ec4ae9a232c2e3887dc
This patch set adds a relative symlink from docs/source/releasenotes
to releasenotes/source/ so that releasenotes can be published
to deckhand.readthedocs.io. A new tab link is added on the
documentation page which references Deckhand's release notes.
The command used to generate the symlink was:
ln -rs releasenotes/source/ docs/source/releasenotes
Change-Id: I07058ca47ecc7b1fb211cae53aedc5f796542cff
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.
Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.
Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
Add a blurb to the replacement documentation that differentiates
the layering replace action from document replacement to avoid
conflation of the terms.
Change-Id: Ie3da4645c5fa4ed0eab95184443acbb01c5b2740
Recently https://review.gerrithub.io/#/c/406626/ broke functional
tests via uwsgi because it changed how entrypoint.sh is called
which is performed during functional-tests.sh when uwsgi is used
to drive the server for testing. This changes how entrypoint.sh
is called so that the tests now pass with uwsgi.
Change-Id: I8252350676e61d5214da11e9ed282cc3399288d9
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.
Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.
Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
Follow up to [0] which includes document replacement
documentation. Includes documentation on the following
subjects:
* document replacement (new)
* document abstraction (new)
* document parent selection (edit)
This PS also renames some docs files by replacing "_"
with "-" in files with compound names for consistency.
[0] https://review.gerrithub.io/#/c/403888/
Change-Id: I6c1ba1e77347c266a6a9d471c9d7a747b1cef6eb
Under load, Deckhand will fail liveness checks with a 1 second timeout.
This Patchset extends the timout to 10 seconds and spaces the period
between checks to 20 seconds.
Adds labels to keystone user job.
Change-Id: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c
This PS integrates document replacement with document layering. The case
works something like this:
GIVEN:
- Parent A
- Child B
- Child C
WHEN:
- Child B is a replacement for A
THEN:
- B must layer with A, then C must layer with B, rather than A,
as B replaces A.
This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.
To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.
Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.
The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.
Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
This PS tests that Deckhand accepts and parses documents that use YAML
anchors and pointers. The restriction is that this be used intra-document
as anchors and pointers can't be used across documents.
Change-Id: I28e502a46e5fbb8cc27cf60e83d9b9e9572a3d36
This is a trivial PS that fixes the tox -e cover job in
tox.ini which was recently broken with [0].
[0] https://review.gerrithub.io/#/c/405318/
Change-Id: Id50a6348e6f306c3d8d68fdd79eb331880e7498b
Changes the entrypoint.sh options for uwsgi to include:
-b 32768 : for larger header/url handling
--die-on-term : for more 'normal' handling of SIGTERM
--lazy-apps : to delay init of python until after forking workers
--master : to provide a master process for handling request dispatch
The purpose of these changes is intended to avoid some crash behavior
that is occuring when the process being forked has an open db connection.
The --lazy-apps option should delay initialization. The other options are
recommended by uwsgi documentation, specicially the --master option.
The larger buffer size is not strictly recommended, but matters when large
headers are included.
The die-on-term option should provide better behavior in the container
environment.
Related-Change: I60adeffff5461fdda957124232bc5a606baae413
Change-Id: I70510246576a8fb6aa216e7c9c7e97c1c9ab791c
Layering code was not using a parse cache for jsonpath
This change adds use of the cache around all calls to jsonpath_ng.parse
Change-Id: I800eb397badf19ed2ea47b88fa7c91e4a09225ef