404 Commits

Author SHA1 Message Date
Felipe Monteiro
3d53d53712 Add uwsgi functional test check to .zuul.yaml
This adds a uwsgi functional test check to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.

Change-Id: I931ab4d11719daca7665d3a25b00e353c707237e
2018-05-29 20:25:15 +00:00
Felipe Monteiro
119080b597 Use Ansible playbooks for functional testing gating
This patchset converts much of the previous logic in
functional-tests.sh into Ansible playbooks to be executed
by Zuul. This mainly includes all the Docker-related
deployment logic.

The functional-tests.sh script has been slimmed down to
just work with uwsgi so that a standalone functional
test deployment can be performed relatively easily,
mainly by developers.

Finally, py27 support for the gate has been dropped
as the Dockerfile in this project currently assumes
python3 for installing requirements and so forth,
leading to requirements issues blocking the gate.

Change-Id: I903a2845390061641d292fb0c016ba6a53723fc9
2018-05-29 15:17:28 +00:00
Zuul
53a5145200 Merge "Drop gather prom metrics from airship-deckhand-ubuntu job" 2018-05-25 21:14:25 +00:00
Felipe Monteiro
5cd77c8a22 Drop gather prom metrics from airship-deckhand-ubuntu job
Prometheus scrapers aren't running in the airship-deckhand
deployment job, so no need to include this here. It is
mainly being removed because of a POST_FAILURE issue
resulting from including the role in question in this job.

Change-Id: Ia080ff333c82f5b4b13d2e5db8c88741c0dc4613
2018-05-23 22:55:52 +00:00
melissaml
e86fa5300c fix typos in documentation
Change-Id: Ie0db25163554ce6532503eb998a8f3c7f349df11
2018-05-23 20:59:34 +08:00
Felipe Monteiro
1cbe993b6b Add functional tests to .zuul.yaml
This patchset adds functional tests to .zuul.yaml. Additionally
it adds a functional-py35 job as well which will also be kicked
off via Zuul.

Change-Id: Ic2d1db4d3cd65c4d93c3a6f04e6efeeba9755f07
2018-05-19 05:22:24 +00:00
Zuul
d75977d6c0 Merge "Zuul: Initial Airship-Deckhand checks" 2018-05-17 21:31:00 +00:00
Felipe Monteiro
e1f22a76c7 Update .gitreview for openstack infra
Change-Id: Ie77a8bafd817a352ee8d3f10770783cd6870c3ce
2018-05-17 19:21:56 +01:00
Pete Birley
7a32dc92e0 Zuul: Initial Airship-Deckhand checks
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.

Change-Id: If5b6550c6f0d9d1f96c4822e72d7d519dfca3c79
Signed-off-by: Pete Birley <pete@port.direct>
2018-05-17 12:36:19 -05:00
Felipe Monteiro
177675e96f [fix] Parent substitution/layering before replacement
Currently it doesn't seem document replacement works
exactly as expected: The parent-replacement document
can receive layering and substitution data prior to
being replaced. Currently, Deckhand does not account
for this scenario.

A child-replacement depends on its parent-replacement
the same way any child depends on its parent: so that the
child layers with its parent only after the parent has
received all layering and substitution data. But other
documents that depend on the parent-replacement actually
depend on the child-replacement instead as the
child-replacement replaces its parent. So the dependency
chain is: PR -> CR -> anything that layers with PR.

A unit and functional test have been added for regression.

Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I353393f416aa6e441d84add9ebedcd152944d7e8
2018-05-14 19:31:57 -04:00
Anthony Lin
a004c7a19e Update Deckhand API Pod Labels
As part of ongoing effort to update the "application" and
"component" labels for the UCP components, there is a need
to align with the convention. We will update the label for
the deckhand API pod in this case.

Also updated helm_tk.sh to point to openstack-helm-infra for
reference to helm-toolkit as helm-toolkit has been removed
from the openstack-helm repo [0]

[0] https://review.openstack.org/#/c/558065/

Change-Id: I753c4ce653790250b79986c670224d0962f7676f
2018-05-14 04:54:55 +00:00
Felipe Monteiro
e479cba5ff Update Apache LICENSE
This patchset updates the Apache LICENSE to be consistent with
other OpenStack services [0]. This in effect drops the appendix
section not present in the OpenStack license.

https://github.com/openstack/nova/blob/master/LICENSE

Change-Id: I9b350c73d92633f45cbd9c32f9b8740fde3946df
2018-05-10 22:25:14 +01:00
Felipe Monteiro
111018ce24 [chart] Remove liveness probe to stop DH pod from being killed
This is to stop the DH pod from being killed in production whenever
DH receives multiple concurrent requests from another service,
causing all its threads to become occupied with servicing those
requests, causing the liveness probe to fail, causing the DH pod
to be killed. This is highly undesirable and as a temporary
workaround we will drop the liveness probe altogether.

This partially reverts I1a1c107706862431e53668a864db622499e63c6f
Additional reading: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c

Change-Id: Ic81c0c1d6e3cd3ab3b326054b9c882962d240968
2018-05-09 14:48:23 +00:00
Felipe Monteiro
93a3274425 Add limit query filter param
This PS adds a limit query filter parameter to allow users to limit
the number of documents returned by revision documents as well
as rendered documents.

Change-Id: Ic15dc59cd21d82be552fa7b9885754bde47724a0
2018-05-09 01:17:06 +00:00
Felipe Monteiro
97578a933f [fix gate] Fix pep8 errors
Fix failing pep8 errors which were never being flagged but now are,
possibly due to changes in flake8 rules. This patchset corrects
the following errors:

./deckhand/engine/layering.py:567:21: W503 line break before binary operator
./deckhand/engine/secrets_manager.py:406:33: W503 line break before binary operator
./deckhand/engine/utils.py:33:17: W503 line break before binary operator
./deckhand/common/utils.py:292:17: W503 line break before binary operator

Change-Id: Ic26aecb6b8049e138a826af9953f45298e817795
2018-05-09 02:14:08 +01:00
Luna Das
8538ff5671 Add no oauth middleware to bypass keystone authentication
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.

The logic is similar to Drydock's here: [0].

[0] 1c78477e95/drydock_provisioner/util.py (L43)

Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
2018-05-08 03:46:52 +01:00
Felipe Monteiro
444e4d9dcc [fix gate] Unblock failing integration job
This PS fixes the integration job failing on the following:

  Unable to match $.[0].data.`split(:, 0, 1)` + "://" +
  $.[0].data.`split(/, 2, 3)` + "/v1" as
  http://barbican.openstack.svc.cluster.local:80/,
  got http://barbican.openstack.svc.cluster.local:80/v1

This change corrects this issue.

Change-Id: Iccc172efa37fe06df3af59a569bbc6f135c773fe
2018-05-07 18:05:13 +00:00
Felipe Monteiro
2ae61e1633 [validation] Add validation codes DXXX for validation failures
This patchset basically adds validation error codes (D001, D002)
for validation failures to align with UCP standard. The codes
are as follows:

* D001 - Indicates document sanity-check validation failure pre- or
  post-rendering.
* D002 - Indicates document post-rendering validation failure.

Change-Id: I01a99ec25c214629209ade5181debc39794c5561
2018-05-05 02:08:18 +00:00
Bryan Strassner
1860c8679b Add tests target to Makefile for Deckhand
Change-Id: I9a152c4ebb4c2b9637250720158acb0af5434288
2018-05-04 15:00:11 -05:00
Luna Das
7385d07739 Add single resource substitution feeds multi destinations
This PS adds single resource substitution feeds multiple
destinations.

Closes #21

Change-Id: Ib435d2cad3324bfce1ed8906fe34421a25389bfc
2018-05-03 14:52:35 +00:00
Felipe Monteiro
3cbd29e414 [test] Unskip integration tests
Due to a rebase issue, integration tests are skipping which is
unintended and undesirable. This fixes the issue.

Change-Id: I8e37f4610e4a1e9b8fb4fd81aa8643100cd6dfd5
2018-05-02 17:11:31 -04:00
Felipe Monteiro
c094b16ff6 Clean up integration test script
This PS:

* adds a trap to clean up OSH which is deployed in the
  course of integration tests. It appears as though node cleanup
  in Jenkins is hanging so this is to try to ameliorate that
* creates a deckhand.conf.test to be used by functional and
  integration tests instead of writing it out dynamically [0]
* updates logging.conf.sample to dump logs to stdout/stderr
  by default as this is amenable to containers
* makes test_gabbi.py common between functional and integration
  tests to avoid unnecessary code duplication

[0] review comments in https://review.gerrithub.io/#/c/att-comdev/deckhand/+/407638/

Change-Id: I762fb0bde5f75effcde56316d92bd57b30026995
2018-05-01 21:45:03 +00:00
Felipe Monteiro
6b56f1d1ab Update README to correct typos and deprecated, misleading sections
This PS simply updates the README to correct typos (intgration =>
integration) and reword misleading sections, as well as update
deprecated sections. Finally, new sections related to Barbican
have been added.

Change-Id: I92611cf2aecf5ee1295ba9014002cd0e18f3a6af
2018-04-30 18:42:58 +00:00
Felipe Monteiro
d02e1bcf53 [feature] Endpoint for listing revision validations with details
This patch set adds a new endpoint to the Validations API which allows
for listing all validations for a given revision with details.
The response body for GET /api/v1.0/{revision_id}/validations/detail
looks like:

  ---
  count: 1
  next: null
  prev: null
  results:
    - name: promenade-site-validation
      url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
      status: failure
      createdAt: 2017-07-16T02:03Z
      expiresAfter: null
      expiresAt: null
      errors:
        - documents:
            - schema: promenade/Node/v1
              name: node-document-name
            - schema: promenade/Masters/v1
              name: kubernetes-masters
          message: Node has master role, but not included in cluster masters list.

Note that the Validations API in general is currently missing fields
like url (as well as next and prev references) which will be included
in a follow up.

This will enable Shipyard to avoid performing a quadratic number
of API look ups when querying Deckhand's Validations API: [0].

The policy enforced for this endpoint is deckhand:list_validations.

APIImpact
DocImpact

[0] 06b5e82ea8/shipyard_airflow/control/configdocs/deckhand_client.py (L265)

Change-Id: I827e5f47bffb23fa16ee5c8a705058034633baed
2018-04-29 01:37:38 +00:00
Felipe Monteiro
236e8be530 Add verbose: true to all functional tests
This is to add verbose: true to all functional tests to enable
better test debugging.

Change-Id: I72b0080e3de6d0edb941291e4215eaa633cb627a
2018-04-29 01:33:46 +00:00
Felipe Monteiro
2165e8b396 [test] Cover all secret Deckhand types in integration tests
This patch set accomplishes 2 things:
1) Fixes an issue in Deckhand caused by improperly using the wrong
   Barbican secret type for the PrivateKey Deckhand document type.
2) Tests all Deckhand secret types against Barbican via integration
   testing.

The current error being raised is this: "Provided object does not match
schema 'Secret': u'privatekey' is not one of ['symmetric', 'passphrase',
'private', 'public', 'certificate', 'opaque']."

Change-Id: I8231c87782902850fe0632d0346c567c7481e95f
2018-04-27 14:25:29 -04:00
Felipe Monteiro
a5bb91bc76 [fix] Handles quotes in JSON path for substitution
This is to fix a use case where a subpath like 'filter:authtoken' in
a JSON path like ".values.conf.paste.'filter:authtoken'.password" fails
because the resulting Python object that is created in memory
for substitution constructs a dict key like "'filter:authtoken'" resulting
in Deckhand failing to index into the dict as the key needs to be
stripped of start and end quotes that are only necessary for achieving
valid YAML syntax.

A unit test is added for regression.

Change-Id: I19974efc977b0cdc5793e649fa068d1a3bd7339e
2018-04-27 01:38:56 +00:00
Felipe Monteiro
e65710bf1a Make Deckhand validation exceptions adhere to UCP standard
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].

To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.

In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.

TODO (in a follow up):

  * Allow 'info_list' to specified for any DeckhandException
    instance.
  * Pass the 'reason' and 'error_list' and etc. arguments to
    all instances of DeckhandException that are raised.

[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure

Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
2018-04-26 18:51:08 +00:00
Neeldhwaj Pathak
4d3f8b5dcd Add .idea/ to gitignore
Change-Id: I565d128d739215bc3dbdd1ca561757be52c5ab07
2018-04-25 16:33:52 +00:00
Felipe Monteiro
5c078ce81c Update releasenotes/docs tox jobs
This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.

Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
2018-04-24 22:34:49 +01:00
Felipe Monteiro
1566b9541a Clean up functional test directory and entrypoint script
This PS simply reorganizes Deckhand's functional test directory
to make it more maintainable and readable as right now it is
hard to figure out what is covered by a functional test and
what isn't.

Additionally, the entrypoint for these tests in tools/functional-tests.sh
has also been refactored slightly.

Change-Id: I262c7e1f7cbce248c12ee013a9bab4e32b89adee
2018-04-20 22:07:04 +01:00
Anthony Lin
c29ad4406b Change name of Deckhand Container
We will align the name with the rest of the UCP components, i.e.
change it from 'deckhand' to 'deckhand-api'

Change-Id: I4c65ac1e6371ffa80fd8b42cbe979d71b93e99c7
2018-04-19 23:25:20 -04:00
Felipe Monteiro
f30484a14c Add integration tests
This patch set adds integration tests to Deckhand
where "integration" means the interaction between
Deckhand, Barbican and Keystone. OSH is used to
deploy Keystone and Barbican and Docker to deploy
PostgreSQL and Deckhand.

Unlike functional testing in Deckhand, all
integration tests use the default in-code policy
defaults and an admin token supplied by keystone
to validate authN and authZ.

The test scenarios consist of Deckhand secret
lifecycle management as well as document rendering
with secrets retrieved from Barbican.

Change-Id: Ib5ae1b345b2a4bd579671ec4ae9a232c2e3887dc
2018-04-18 09:05:04 -04:00
Felipe Monteiro
d32c7a2c8d [docs] Publish releasenotes alongside docs to readthedocs
This patch set adds a relative symlink from docs/source/releasenotes
to releasenotes/source/ so that releasenotes can be published
to deckhand.readthedocs.io. A new tab link is added on the
documentation page which references Deckhand's release notes.

The command used to generate the symlink was:

    ln -rs releasenotes/source/ docs/source/releasenotes

Change-Id: I07058ca47ecc7b1fb211cae53aedc5f796542cff
2018-04-15 14:09:06 -04:00
Felipe Monteiro
106038d3cd [fix] Pass secret URI instead of UUID to barbican get_secret
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.

While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].

API ref:

GET /v1/secrets/{uuid}
Headers:
    Accept: application/json
    X-Auth-Token: {token}
    (or X-Project-Id: {project_id})

CLI ref:

$  barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
                           [--max-width <integer>] [--prefix PREFIX]
                           [--decrypt] [--payload]
                           [--payload_content_type PAYLOAD_CONTENT_TYPE]
                           URI

Retrieve a secret by providing its URI.

Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.

[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get

Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
2018-04-11 17:33:58 -04:00
Mark Burnett
7086973386 Add negative functional test for substitution
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.

Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.

Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
2018-04-11 14:04:22 -04:00
Felipe Monteiro
6b9cd17e98 docs: Distinguish replace layering action from document replacement
Add a blurb to the replacement documentation that differentiates
the layering replace action from document replacement to avoid
conflation of the terms.

Change-Id: Ie3da4645c5fa4ed0eab95184443acbb01c5b2740
2018-04-10 21:52:00 -04:00
Felipe Monteiro
e23f46c152 Fix running functional tests via uwsgi
Recently https://review.gerrithub.io/#/c/406626/ broke functional
tests via uwsgi because it changed how entrypoint.sh is called
which is performed during functional-tests.sh when uwsgi is used
to drive the server for testing. This changes how entrypoint.sh
is called so that the tests now pass with uwsgi.

Change-Id: I8252350676e61d5214da11e9ed282cc3399288d9
2018-04-10 16:48:03 -04:00
Felipe Monteiro
a3d32c3459 Raise exception on unfound secret in source document
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.

Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
2018-04-09 17:04:58 +01:00
Felipe Monteiro
35f15ee601 [fix] Drop deckhand.conf from default DECKHAND_CONF_DIR path
This drops deckhand.conf from default DECKHAND_CONF_DIR path so
that:

exec uwsgi \
    ...
    --pyargv "--config-file ${DECKHAND_CONFIG_DIR}/deckhand.conf" \
    ...

Doesn't append deckhand.conf again, resulting in a broken path like:
/etc/deckhand/deckhand.conf/deckhand.conf

Change-Id: I1c9d5ad283fed7158ba70b84b5b4afb27a3ab5d9
2018-04-07 00:20:47 -04:00
Bryan Strassner
5f1fbbee3c [396582] Add alembic support to Deckhand
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.

Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
2018-04-06 23:30:16 -04:00
Scott Hussey
4d90257372 [Fix] Multidigit array index
- Allow array index in substitution destinations to have
  multiple digits

Change-Id: I8ef6241763dd7d841e25774fa041f7f4a3b11c7b
2018-04-05 16:56:30 -05:00
Felipe Monteiro
24b4cf0420 Document replacement documentation
Follow up to [0] which includes document replacement
documentation. Includes documentation on the following
subjects:

  * document replacement (new)
  * document abstraction (new)
  * document parent selection (edit)

This PS also renames some docs files by replacing "_"
with "-" in files with compound names for consistency.

[0] https://review.gerrithub.io/#/c/403888/

Change-Id: I6c1ba1e77347c266a6a9d471c9d7a747b1cef6eb
2018-04-05 15:09:51 -04:00
Bryan Strassner
c962eeb975 [fix] Extend liveness and readiness check times
Under load, Deckhand will fail liveness checks with a 1 second timeout.
This Patchset extends the timout to 10 seconds and spaces the period
between checks to 20 seconds.
Adds labels to keystone user job.

Change-Id: Id2d4deaaf8bf73d6df4639810e6dee3acf79b05c
2018-04-04 15:31:46 -05:00
Felipe Monteiro
74528a518d Document replacement: Layering dependency integration
This PS integrates document replacement with document layering. The case
works something like this:

  GIVEN:
  - Parent A
  - Child B
  - Child C

  WHEN:
  - Child B is a replacement for A

  THEN:
  - B must layer with A, then C must layer with B, rather than A,
    as B replaces A.

This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.

To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.

Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.

The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.

Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
2018-04-04 10:58:28 -04:00
Felipe Monteiro
45441b6618 Test that Deckhand works with YAML anchors/pointers
This PS tests that Deckhand accepts and parses documents that use YAML
anchors and pointers. The restriction is that this be used intra-document
as anchors and pointers can't be used across documents.

Change-Id: I28e502a46e5fbb8cc27cf60e83d9b9e9572a3d36
2018-04-03 16:22:58 -04:00
Felipe Monteiro
4a586e370e Remove unused functions from DB module
This is a trivial PS that removes 2 unused functions from the
DB module.

Change-Id: I5c84d0b9d7b4c7e64a3608dcc5cc5262e0b9424a
2018-04-03 14:42:48 -04:00
Felipe Monteiro
6a521e2eb9 Trivial fix: Fix coverage tox.ini job
This is a trivial PS that fixes the tox -e cover job in
tox.ini which was recently broken with [0].

[0] https://review.gerrithub.io/#/c/405318/

Change-Id: Id50a6348e6f306c3d8d68fdd79eb331880e7498b
2018-04-03 14:17:58 -04:00
Bryan Strassner
4d642f849a [fix] Add uwsgi entrypoint options
Changes the entrypoint.sh options for uwsgi to include:
 -b 32768 : for larger header/url handling
 --die-on-term : for more 'normal'  handling of SIGTERM
 --lazy-apps : to delay init of python until after forking workers
 --master : to provide a master process for handling request dispatch

The purpose of these changes is intended to avoid some crash behavior
that is occuring when the process being forked has an open db connection.
The --lazy-apps option should delay initialization. The other options are
recommended by uwsgi documentation, specicially the --master option.
The larger buffer size is not strictly recommended, but matters when large
headers are included.

The die-on-term option should provide better behavior in the container
environment.

Related-Change: I60adeffff5461fdda957124232bc5a606baae413
Change-Id: I70510246576a8fb6aa216e7c9c7e97c1c9ab791c
2018-04-03 09:19:28 -04:00
Bryan Strassner
c50501cc89 [fix] Updates to use cached jsonpath
Layering code was not using a parse cache for jsonpath
This change adds use of the cache around all calls to jsonpath_ng.parse

Change-Id: I800eb397badf19ed2ea47b88fa7c91e4a09225ef
2018-04-02 16:30:06 -05:00