585 Commits

Author SHA1 Message Date
Rick Bartra
60e82b7bd6 Validate additional 'metadata.replacement' scenarios
This patch set adds additional documentation and unit tests
to validate further replacement scenarios.

In particular this commit adds an additional document check that
looks for documents exisitng in different layers that contain the
same name and same schema without any of them having `replacement: true`

Change-Id: I7c033d32a6755f36e609789a748cbc6d4af06bc2
2018-10-30 10:23:14 -04:00
Rick Bartra
88fe773cd7 Fix document is_control method
The document.py `is_control` method incorrectly checks if a document
is a Control document. Per the documentation [0], Control documents
have `metadata.schema` of `metadata/Control/v1`. This commit updates
the `is_control` method to correctly check for Control documents.

[0] 1d4cc81dfa/doc/source/users/document-types.rst (control-documents)

Change-Id: I60ca8f31a61987b4e756784fce0f5a751639ae9e
2018-10-30 09:59:38 -04:00
Zuul
464d2c0ea5 Merge "Validate bucket diffing works with revision rollback" 2018-10-29 17:26:38 +00:00
Zuul
27aeeb8fea Merge "docs: Add documentation on data redaction" 2018-10-29 17:26:37 +00:00
Zuul
eb178e1d7f Merge "refactor: Move replacement checks into separate module" 2018-10-29 17:26:37 +00:00
Zuul
56e606bf4b Merge "fix: Redact secondhand substitutions of sensitive data" 2018-10-29 17:13:25 +00:00
Zuul
475655ac5a Merge "fix: Correct .data path layering edge case" 2018-10-29 14:31:11 +00:00
Zuul
a5deb49e7e Merge "docs: Use sphinx-apidoc library for autodoc compatibility" 2018-10-27 23:00:16 +00:00
Felipe Monteiro
9d91a072cd docs: Use sphinx-apidoc library for autodoc compatibility
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.

More info: https://pypi.org/project/sphinxcontrib-apidoc/

Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
2018-10-27 22:52:39 +01:00
Felipe Monteiro
a2602f3851 fix: Add missing requirements to doc/requirements.txt for RTD
I872b90e03a8f9e3de3fd717ed340b53f095f01a5 tried to fix a autodoc
issue with RTD but in turn introduced an issue causing RTD doc
publish job to fail; this fixes that.

Change-Id: I53fdcb609f8129f61abf8266e05ff49bfbef18b5
2018-10-27 14:26:52 -04:00
Felipe Monteiro
018162f1ef rtd: Fix warnings in RTD causing autodoc to fail
This patch set fixes warnings [0] in RTD which are causing autodoc
to fail to build, resulting in missing documentation as in
here: https://airship-deckhand.readthedocs.io/en/latest/operators/exceptions.html

Example warnings:

WARNING: autodoc: failed to import module 'deckhand.policy'; the following exception was raised:
No module named 'falcon'

[0] https://readthedocs.org/api/v2/build/8001684.txt

Change-Id: I872b90e03a8f9e3de3fd717ed340b53f095f01a5
2018-10-27 15:21:54 +00:00
Felipe Monteiro
47ade1f0da fix: Redact secondhand substitutions of sensitive data
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.

Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
2018-10-25 09:39:50 -04:00
anthony.bellino
7defe473d2 Redact rendered Documents
- Uses the rendered-documents endpoint
- Adds a query parameter ?cleartext-secrets
- Adds unit tests, updates integration tests

Change-Id: I02423b9bf7456008d707b3cd91edc4fc281fa5fc
2018-10-24 22:42:25 -04:00
Zuul
018919ea5c Merge "Fix: adding back the possibility to add arbitrary labels" 2018-10-24 21:50:07 +00:00
Roman Gorshunov
e2edd5fef9 Fix: adding back the possibility to add arbitrary labels
Arbitrary labels could be added as `make` parameter `LABEL=`, which is used in
att-comdev/cicd Jenkins pipelines.

Sample parameter:
'LABEL=org.label-schema.vcs-url=${GERRIT_CHANGE_URL} \
--label org.label-schema.base-image=${base_sha256}'

Sample usage:
See Jenkinsfile files under images/ directory in att-comdev/cicd repo.

In addition to that, if `COMMIT` variable is undefined when invoking `make`, we
use result of `git rev-parse HEAD` command, which should output latest git
commit ID.

[0] https://github.com/att-comdev/cicd

Change-Id: I795d1443f60f362c5ecb244cb90ee8052603854f
2018-10-24 18:28:52 +02:00
Zuul
1d4cc81dfa Merge "docs: Elaborate on document layering in documentation" 2018-10-24 14:03:35 +00:00
Zuul
b4cd48cde0 Merge "trivial: Add missing alembic upgrade head to manual install" 2018-10-24 02:27:40 +00:00
Felipe Monteiro
0d5189e226 trivial: Add missing alembic upgrade head to manual install
This patch set adds `./entrypoint.sh alembic upgrade head` command
to manual install section in Getting Started guide, without which
Deckhand server cannot start correctly.

Change-Id: Id486cc98b8fd93271a8571364adfb9fb30bf4bcc
2018-10-23 22:09:22 -04:00
Zuul
7d697012fc Merge "Redacts Raw Documents" 2018-10-22 15:24:20 +00:00
Felipe Monteiro
24d86ea749 refactor: Move replacement checks into separate module
This patch set refactors replacement validation checks
in Deckhand's layering module into a separate module for
better code organization.

Change-Id: If973148ac8220b96f61128b8a7266e6fd57e76b9
2018-10-20 17:08:16 -04:00
Felipe Monteiro
f711a83ee7 docs: Add documentation on data redaction
This patch set adds documentation related to data redaction in
order to capture the purpose behind it and how it adds an
added layer of security to Deckhand.

Change-Id: Icb28970684a9026fda200273a14d9ba421f627d7
2018-10-20 14:48:19 -04:00
Aaron Sheffield
349e5600df Redacts Raw Documents
- If a document has a storage policy of encrypted
 - Redacts (sha256) the data section.
 - Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
  ?cleartext-secrets=true to show the non-redacted values.

Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
2018-10-19 23:56:12 -05:00
Zuul
a991513eff Merge "fix: Add validation logic to check for duplicate documents in engine" 2018-10-19 21:35:47 +00:00
Zuul
37716a1cdd Merge "chore: Migrate templates from project-config to in-tree" 2018-10-18 22:03:10 +00:00
Felipe Monteiro
035841416b Validate bucket diffing works with revision rollback
Adds a unit test to validate following scenario:

1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)

Validate that diffing works for rolled-back revision.
All cases above use same bucket.

Also refactors some test logic for neatness.

Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
2018-10-18 19:07:42 +01:00
Felipe Monteiro
b80df59d11 fix: Address small issues with revision rollback controller
1. There is no exception called `InvalidRollback` in Deckhand (it
was removed a while back). Instead, the only exception that
db_api.revision_rollback raises is RevisionNotFound from
the revision_get call internally.

So catch that instead from the controller.

2. The default value of parameters is `str` so when revision_id
of '0' is passed to the db module for processing, it skips over
the check for `if revision_id == 0` as revision_id is a str,
not int. So this leverages builtin int converter logic in
falcon [0] but requires uplifting the version of falcon to
at least 1.3.0 to make use of it [1].

[0] https://falcon.readthedocs.io/en/stable/api/routing.html#field-converters
[1] https://falcon.readthedocs.io/en/1.3.0/api/routing.html#field-converters

Change-Id: I068cd9e9b6818a5d51501f2718ee2d40d556c094
2018-10-18 09:45:16 -04:00
Felipe Monteiro
fb08d1fd5f chore: Migrate templates from project-config to in-tree
This patch set moves remaining Zuul templates from project-config
to Deckhand to satisfy the following requirement:

https://docs.openstack.org/infra/manual/zuulv3.html#where-should-jobs-and-templates-live

Change-Id: Ia10a8c7fa34ab130a01f834bd22c947b980a22a3
Needed-By: Iaf2e0e1fd79b68ce1ea607f19cff5c40e47b05b1
2018-10-18 05:30:50 +00:00
Felipe Monteiro
38c1d7a1a3 fix: Pin down Deckhand package requirements
This patch set pins down package requirements for the following
reasons:

* Inconsistencies between ranges and direct pins
* Deckhand isn't gated against requirements repo so changes to
  some packages isn't actively tested
* Other Airship components use pins, so Deckhand might break them
  when package requirements go out of sync between components
* Deckhand as of yet has no stable release so Airship relies on
  master for stability so it is better to ensure stability over
  anything else

Change-Id: I20ef4be3e01cd267771098d33447ccb61aee20b4
2018-10-18 02:36:35 +01:00
Zuul
bd9e7c7e96 Merge "optimization: Skip post-validation for rendered document cache hit" 2018-10-16 16:30:32 +00:00
Zuul
40301cc3ba Merge "Adding image tags on every commit" 2018-10-16 13:43:49 +00:00
Zuul
12fe505b05 Merge "add python 3.6 unit test job" 2018-10-15 00:26:50 +00:00
Prateek Dodda
2c4c5a9c63 fix: Add validation logic to check for duplicate documents in engine
This patch set adds validation logic to document_validation.py (in
Deckhand's engine module) so that components that rely on
Deckhand's engine for document rendering (such as Promenade
or Pegleg) can fail fast when they provide Deckhand with a
duplicate document. Must pass pre_validate=True to layering
module which currently is the case for Promenade, et. al.

Before this change, Deckand only supported this logic at
the DB level (requiring service instantion); this is now no longer
the case.

Change-Id: I6d1c8214775aa0f3b5efb1049972cf847f74585b
2018-10-11 22:33:01 +00:00
Roman Gorshunov
9dcc0ae937 Adding image tags on every commit
Added container image tagging on quay.io with commit id on every merge,
when certain files but not the code are changed (docs, charts, etc.).

Previously we were building updated container images, adding tags and
uploading images onto quay.io only when the code has been changed. The
main reason for not re-building images for every single merged commit
was a long waiting time in Zuul `post` queue for a node (VM)
assignment for the job to run on.

With this change we will have image tag for every merged commit,
including documentation commits, and those tags would be added almost
immediately (within ~1 min), as job runs directly on Zuul and does not
wait in Zuul queue for a node (VM) assignment.

Change-Id: I9692947b74a16de0eb3e8100d30cd4310069abfe
2018-10-11 09:04:21 +02:00
Felipe Monteiro
1ac9abb555 docs: Elaborate on document layering in documentation
This patchset elaborates on document layering in the documentation
to provide much greater clarity into what layering is and its
associated concepts, including: layer, layer order, layering policy,
layering definition, document abstraction, parent selection,
layering actions, etc.

Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906
2018-10-10 19:06:38 +00:00
Zuul
0462b7b929 Merge "Add explicit start/end to Deckhand response middleware" 2018-10-08 14:48:23 +00:00
Felipe Monteiro
2ea808cae2 fix: Correct .data path layering edge case
This patch set corrects logic for an edge case in layering where
the action `path` is set to `.data`. In this case this means
that the root of the data section should be used, i.e. '.'
or '$.'. The previous adjustment was incorrect: .data was being
changed to empty string ''. This fixes that logic to change to
'.'.

Change-Id: Id6cf0d4d65020220c540eb162a33055035336cde
2018-10-07 15:10:09 -04:00
Zuul
f9e4b5993f Merge "[Gate Fix] Fix failing functional/integration tests" 2018-10-05 21:17:39 +00:00
Zuul
fd768bc163 Merge "docs: Reorganize documentation structure" 2018-10-05 20:49:07 +00:00
Felipe Monteiro
88e1c12b23 Add explicit start/end to Deckhand response middleware
Pegleg has linting rules dedicated to checking for explicit starts
and so on, so it makes sense that Deckhand just adds this in for
every response as it is a nice feature that better delineates
starting and endpoints points for individual YAML documents.

Change-Id: I6324cfa268ddf250a9c78cb663e7015a171bbc19
Related-Change: https://review.openstack.org/#/c/604123
2018-10-05 18:07:45 +00:00
Felipe Monteiro
3f39e639ea [Gate Fix] Fix failing functional/integration tests
This patch set pins oslo.messaging to the last stable version
to fix the gate [0]. 9.0.0 is incompatible with current
requirements.

[0] https://pypi.org/project/oslo.messaging/#history

Change-Id: I6cf3e9f33ac3efcbf48675273899af38fef86e91
2018-10-05 18:18:55 +01:00
Felipe Monteiro
18ae85a229 optimization: Skip post-validation for rendered document cache hit
This patch set adds logic to optimize performance around rendered
documents caching further by implementing logic to skip over
post-rendering validation when cache hits occur. This works because:

* if the result set isn't cached, then post-validation is performed;
  if the post-validation fails the just-cached result set is
  invalidated
* if the result set is cached, then post-validation for that
  result set has already been performed successfully, so it
  can be safely skipped over

It was discovered via profiler anaylsis that document validation
accounts for an appreciable amount of run time, so optimizing
around this makes a noticeable difference.

Change-Id: I18344ea750cf1028a377028f80ecdd931d9672c0
2018-10-02 18:58:07 -05:00
Felipe Monteiro
ad376dd3b3 trivial: Fix README documentation badge
This patch set fixes the README badge which isn't rendering
correctly (e.g. [0]). Thus it has been changed to [1]. Also
adds image build status via a badge, similar to Armada [2].

[0] https://github.com/openstack/airship-deckhand
[1] https://readthedocs.org/projects/airship-deckhand/badge/?version=latest
[2] https://raw.githubusercontent.com/openstack/airship-armada/master/README.rst

Change-Id: Id8cbf63e5714576d3fe9f209947b04459a3475f4
2018-10-02 18:32:42 -05:00
Felipe Monteiro
a8660a7e53 docs: Reorganize documentation structure
This patch set reorganizes Deckhand's documentation structure
for better organization into 3 distinct categories:

* developer's guide
* operator's guide
* user's guide

This means that the RTD navigation menu on the left-hand side
will have fewer links (see list above) making navigation much
easier. This is similar to how Armada organizes its documentation
too.

This patch set also updates README section with a better
overview and trims some fat from it (remove testing
documentation as it doesn't really belong there -- there
is a dedicated page for that already).

Finally, this patch set changes the exceptions page to
render as a basic list of autoexception classes because
the current tabularized view is not rendering correctly
on RTD [0].

[0] https://airship-deckhand.readthedocs.io/en/latest/exceptions.html
Change-Id: I162383bf8e3bbd5004603c979ac7b0d760a210c4
2018-09-26 20:29:02 -04:00
Zuul
243c2741bd Merge "Fix: various documentation and URL fixes" 2018-09-26 20:21:50 +00:00
Roman Gorshunov
d41e5a44ca Fix: various documentation and URL fixes
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes

Change-Id: I12b2fa8fbec37a483a0ad50382e08f51ed97533a
2018-09-25 11:44:57 +02:00
Zuul
84de5c17b8 Merge "Fix: git commit id labels on images" 2018-09-21 20:25:16 +00:00
Roman Gorshunov
48f79aee44 Fix: git commit id labels on images
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
   or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
   a few other properties (annotations) according to the latest Specs

Change-Id: Iebb37edd003204d3adc41aa9af76612ab419993a
2018-09-21 03:31:12 +02:00
zhangdebo
0b5aa2e98a Replace Chinese quotes with English quotes
Change-Id: I323c50a0cea3dd1e5cfa8180ab46d9af4c610f94
2018-09-20 07:29:27 -04:00
Zuul
7dddbd56d4 Merge "substitution: Recursive pattern replacement" 2018-09-19 21:38:08 +00:00
Zuul
aff835704e Merge "trivial: Fix error message for non-matching policy checks" 2018-09-19 20:27:08 +00:00