Browse Source

Validate existence of "deployment-version" doc on configdocs commit

This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
changes/30/656530/9
Carter, Matthew (mc981n) 2 years ago
parent
commit
64171aacf4
  1. 9
      charts/shipyard/values.yaml
  2. 1
      doc/source/API.rst
  3. 25
      doc/source/_static/shipyard.conf.sample
  4. 41
      doc/source/site-definition-documents.rst
  5. 25
      src/bin/shipyard_airflow/etc/shipyard/shipyard.conf.sample
  6. 32
      src/bin/shipyard_airflow/shipyard_airflow/conf/config.py
  7. 8
      src/bin/shipyard_airflow/shipyard_airflow/control/validators/validate_deployment_configuration.py
  8. 45
      src/bin/shipyard_airflow/shipyard_airflow/control/validators/validate_deployment_version.py
  9. 2
      src/bin/shipyard_airflow/tests/unit/control/test_action_validators.py
  10. 6
      tools/resources/shipyard.conf

9
charts/shipyard/values.yaml

@ -416,6 +416,10 @@ conf:
# deny_all rule is hard coded in the policy.py code to allow no access.
policy_default_rule: deny_all
document_info:
# The name of the deployment version document that Shipyard validates
deployment_version_name: deployment-version
# The schema of the deployment version document that Shipyard validates
deployment_version_schema: pegleg/DeploymentData/v1
# The name of the deployment configuration document that Shipyard expects
# and validates
deployment_configuration_name: deployment-configuration
@ -425,6 +429,11 @@ conf:
# The schema of the deployment strategy document that Shipyard expects
# and validates.
deployment_strategy_schema: shipyard/DeploymentStrategy/v1
validations:
# Control the severity of the deployment-version document validation
# that Shipyard performs during commit configdocs.
# Possible values are Skip, Info, Warning, and Error
deployment_version_commit: Skip
airflow_config_file:
path: /usr/local/airflow/airflow.cfg
airflow:

1
doc/source/API.rst

@ -203,6 +203,7 @@ Responses
with no contents.
- The request has no new/changed contents for the collection.
- The request is missing a Content-Length header.
- The provided document(s) fail Shipyard/Deckhand validations.
409 Conflict
A condition in the system is blocking this document ingestion

25
doc/source/_static/shipyard.conf.sample

@ -90,6 +90,14 @@
# From shipyard_api
#
# The name of the deployment version document that Shipyard validates (string
# value)
#deployment_version_name = deployment-version
# The schema of the deployment version document that Shipyard validates (string
# value)
#deployment_version_schema = pegleg/DeploymentData/v1
# The name of the deployment-configuration document that Shipyard expects and
# validates (string value)
#deployment_configuration_name = deployment-configuration
@ -425,3 +433,20 @@
# type is used to perform the service lookup in the Keystone service catalog.
# (string value)
#service_type = shipyard
[validations]
#
# From shipyard_api
#
# Control the severity of the deployment-version validation validation during
# commit configdocs. (string value)
# Possible values:
# Skip - Skip the validation altogether
# Info - Print an Info level message if the validation fails
# Warning - Print a Warning level message if the validation fails
# Error - Return an error when the validation fails and prevent the commit from
# proceeding
#deployment_version_commit = Skip

41
doc/source/site-definition-documents.rst

@ -38,7 +38,7 @@ of the Armada manifest that will be used during the deployment/update.
A `sample deployment-configuration`_ shows a completely specified example.
Note that the name and schema Shipyard expects the deployment configuration
document to have is conifgurable via the document_info section in the
document to have is configurable via the document_info section in the
:ref:`Shipyard configuration <sample-configuration>`, but should be left
defaulted in most cases.
@ -185,7 +185,7 @@ document for the site. Example::
the group a success.
Note that the schema Shipyard expects the deployment strategy document to have
is conifgurable via the document_info section in the
is configurable via the document_info section in the
:ref:`Shipyard configuration <sample-configuration>`, but should be left
defaulted in most cases.
@ -625,6 +625,43 @@ An Example of CLI `describe action` command output, with failed processing::
step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_nodes 6 failed
Deployment Version
-------------------
A deployment version document is a Pegleg_-generated document that captures
information about the repositories used to generate the site defintion. The
presence of this document is optional by default, but Shipyard can be
:ref:`configured <sample-configuration>` to ensure this document exists, and
issue a warning or error if it is absent from a configdocs collection.
Document example::
---
schema: pegleg/DeploymentData/v1
metadata:
schema: metadata/Document/v1
name: deployment-version
layeringDefinition:
abstract: false
layer: global
storagePolicy: cleartext
data:
documents:
site-repository:
commit: 37260deff6a213e30897fc284a993c791336a99d
tag: master
dirty: false
repository-of-secrets:
commit: 23e7265aee4843301807d649036f8e860fda0cda
tag: master
dirty: false
Currently, Shipyard does not use this document for anything. Use of this
document's data will be added to a future version of Shipyard/Airship.
Note, the name and schema Shipyard expects this document to have can be
configured via the document_info section in the
:ref:`Shipyard configuration <sample-configuration>`.
.. _Pegleg: https://git.airshipit.org/cgit/airship-pegleg
.. _`Armada manifest document`: https://airship-armada.readthedocs.io/en/latest/operations/guide-build-armada-yaml.html?highlight=manifest
.. _`Default configuration values`: https://git.airshipit.org/cgit/airship-shipyard/tree/src/bin/shipyard_airflow/shipyard_airflow/plugins/deployment_configuration_operator.py
.. _DeploymentConfiguration: https://git.airshipit.org/cgit/airship-shipyard/tree/src/bin/shipyard_airflow/shipyard_airflow/schemas/deploymentConfiguration.yaml

25
src/bin/shipyard_airflow/etc/shipyard/shipyard.conf.sample

@ -90,6 +90,14 @@
# From shipyard_api
#
# The name of the deployment version document that Shipyard validates (string
# value)
#deployment_version_name = deployment-version
# The schema of the deployment version document that Shipyard validates (string
# value)
#deployment_version_schema = pegleg/DeploymentData/v1
# The name of the deployment-configuration document that Shipyard expects and
# validates (string value)
#deployment_configuration_name = deployment-configuration
@ -425,3 +433,20 @@
# type is used to perform the service lookup in the Keystone service catalog.
# (string value)
#service_type = shipyard
[validations]
#
# From shipyard_api
#
# Control the severity of the deployment-version validation validation during
# commit configdocs. (string value)
# Possible values:
# Skip - Skip the validation altogether
# Info - Print an Info level message if the validation fails
# Warning - Print a Warning level message if the validation fails
# Error - Return an error when the validation fails and prevent the commit from
# proceeding
#deployment_version_commit = Skip

32
src/bin/shipyard_airflow/shipyard_airflow/conf/config.py

@ -291,6 +291,18 @@ SECTIONS = [
title=('Information about some of the documents Shipyard needs to '
'handle'),
options=[
cfg.StrOpt(
'deployment_version_name',
default='deployment-version',
help=('The name of the deployment version document that '
'Shipyard validates')
),
cfg.StrOpt(
'deployment_version_schema',
default='pegleg/DeploymentData/v1',
help=('The schema of the deployment version document that '
'Shipyard validates')
),
cfg.StrOpt(
'deployment_configuration_name',
default='deployment-configuration',
@ -314,6 +326,26 @@ SECTIONS = [
),
]
),
ConfigSection(
name='validations',
title='Validation Configurations',
options=[
cfg.StrOpt(
'deployment_version_commit',
default='Skip',
help=('Control the severity of the deployment-version '
'validation validation during commit configdocs. '),
ignore_case=True,
choices=[('Skip', 'Skip the validation altogether'),
('Info', 'Print an Info level message if the '
'validation fails'),
('Warning', 'Print a Warning level message if the '
'validation fails'),
('Error', 'Return an error when the validation fails '
'and prevent the commit from proceeding')]
),
]
),
]

8
src/bin/shipyard_airflow/shipyard_airflow/control/validators/validate_deployment_configuration.py

@ -24,6 +24,7 @@ from shipyard_airflow.common.document_validators.document_validator import (
DocumentValidator
)
from .validate_deployment_strategy import ValidateDeploymentStrategy
from .validate_deployment_version import ValidateDeploymentVersion
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
@ -50,7 +51,7 @@ class ValidateDeploymentConfigurationFull(
ValidateDeploymentConfigurationBasic):
"""Validates the DeploymentConfiguration
Includes a triggered check for DeploymentStrategy
Includes triggered checks for DeploymentStrategy and DeploymentVersion
"""
def do_validate(self):
try:
@ -76,4 +77,9 @@ class ValidateDeploymentConfigurationFull(
"'all-at-once' is assumed, and deployment strategy will "
"not be further validated")
if CONF.validations.deployment_version_commit.lower() != 'skip':
self.add_triggered_validation(
ValidateDeploymentVersion,
CONF.document_info.deployment_version_name)
super().do_validate()

45
src/bin/shipyard_airflow/shipyard_airflow/control/validators/validate_deployment_version.py

@ -0,0 +1,45 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Classes and functions to support Shipyard specific document validation
Set up a validator for the DeploymentData document generated by pegleg
Only validates that which is not already covered by schema validation, which
is performed by Deckhand on Shipyard's behalf.
"""
from shipyard_airflow.common.document_validators.document_validator import (
DocumentValidator
)
import logging
from oslo_config import cfg
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
class ValidateDeploymentVersion(DocumentValidator):
"""Validates the existence of the deployment data document"""
def __init__(self, **kwargs):
super().__init__(**kwargs)
schema = CONF.document_info.deployment_version_schema
# Just capitalize the "missing_severity", and then the base class will take
# care of whether or not the value is actually valid
missing_severity = CONF.validations.deployment_version_commit.capitalize()
def do_validate(self):
# Need to define this since it is abstract in the base class, but all
# we need to check for this document is its existence, which is taken
# care of by the validate() function in the base class
pass

2
src/bin/shipyard_airflow/tests/unit/control/test_action_validators.py

@ -62,6 +62,8 @@ def get_doc_returner(style, ds_name):
return [dc]
elif doc == 'dep-strat':
return [strategy]
elif doc == 'deployment-version':
return [MagicMock()]
return []
return doc_returner

6
tools/resources/shipyard.conf

@ -48,4 +48,8 @@ policy_default_rule = deny_all
[document_info]
deployment_configuration_name = deployment-configuration
deployment_configuration_schema = shipyard/DeploymentConfiguration/v1
deployment_strategy_schema = shipyard/DeploymentStrategy/v1
deployment_strategy_schema = shipyard/DeploymentStrategy/v1
deployment_version_name = deployment-version
deployment_version_schema = pegleg/DeploymentData/v1
[validations]
deployment_version_commit=Skip
Loading…
Cancel
Save