Commit Graph

152 Commits (02533cb08594bb4add62dc10fa6c22a51b1fe75c)

Author SHA1 Message Date
Michael Beaver 02533cb085 Add unit tests and fix comments/samples
Follow up change for deployment-status ConfigMap [0] adding a few
things that were either missed or need to be added, especially unit
tests for the new DeploymentStatusOperator class


Change-Id: Ieaf55dc7725ea92062fde7997a676726b2396253
2019-05-29 15:02:59 -05:00
Michael Beaver 53e863954b Store status of deployment in a ConfigMap
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps

This operator has been added to the deploy_site, update_site, and
update_software DAGs.

Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
2019-05-28 08:32:07 -05:00
Michael Beaver b5469c39ec Create Xcom Pusher/Puller for concurrency_check
This change adds support in the concurrency_check plugin to push its
status to xcom so other components can easily check if it has passed.
This is especially useful for components that need to always run, but
still want to respect the concurrency check.

This is also a slight refactor of XcomPuller to make it import its dag
names from

Change-Id: I9ca6b43d7789d9499121384d4427835e296c44b8
2019-05-25 19:02:29 -05:00
Carter, Matthew (mc981n) c19580378b Add deployment status generation to Shipyard
This patch set adds functionality to Shipyard's action_helper module
to generate a set of "deployment-status" data. This data can be used
to denote the status and result of a Shipyard action and will be
utilized in a future feature to help track actions/deployments.

This new functionality is not yet being used, and actual Shipyard
functionality should not be altered with this patch set.

Change-Id: I838abfd2144f08708b3fc5a53cc76f124aedff04
2019-05-25 19:02:29 -05:00
Zuul d479d6b969 Merge "Validate existence of "deployment-version" during create configdocs" 2019-05-13 15:39:38 +00:00
Zuul 38c6faab7f Merge "Validate existence of "deployment-version" doc on configdocs commit" 2019-05-13 15:37:02 +00:00
Carter, Matthew (mc981n) c25a78fef4 Update yaml.load to yaml.safe_load in Shipyard unit tests
In all honesty, the warnings that were output about unsafe yaml.load
while running unit tests in Shipyard were just kind of annoying.
They always made me think there was a unit test failure for a split
second. This patch set updates yaml.load to yaml.safe_load in the
configdocs helper unit tests. The unit tests themselves are
effectively unchanged, but now there are no warnings returned.

Change-Id: I65a5d55fbe9eb690ad534a5b7df882762bf5a9fb
2019-05-09 10:34:52 -05:00
Carter, Matthew (mc981n) 0761099337 Validate existence of "deployment-version" during create configdocs
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.

Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.

Change-Id: I754617de81f628a24232e890b12b157ba6731c25
2019-05-09 08:33:52 -05:00
Carter, Matthew (mc981n) 64171aacf4 Validate existence of "deployment-version" doc on configdocs commit
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
2019-05-09 08:25:37 -05:00
Carter, Matthew (mc981n) 222b074cb1 Be configuration driven when referencing document names/schemas
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.

Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
2019-05-02 10:47:38 -05:00
Crank, Daniel (dc6350) 70328e267b Airflow: Update Deckhand / Drydock version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.


Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
2019-04-18 11:50:41 -05:00
Drew Walters cdad9e7cbc airflow: Update Armada version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Armada API client. The Armada client was updated to handle the
additional parameter [1]; however, the Shipyard version Armada uses was
not updated. This change uplifts the version of Armada in
requirements.txt for the Airflow component of Shipyard to a version that
supports the aforementioned, additional, user-context parameter.


Change-Id: I7dfbda842eb136858fbb6d891b96d057a977c913
2019-04-15 12:14:45 -05:00
Smruti Soumitra Khuntia 9c5270b616 User context tracing through logging
This PS adds entry in log for user id and passes on the context
maker to other Airship components from Shipyard during API call.

This will ensure easy tracing of user and context through log

Change-Id: Ib9bfa8f20b641f8bb6c2dca967d9388e30d5735c
2019-04-04 13:19:02 +00:00
Nishant kumar afd2788729 Remove Shipyard queries for Tiller information
Currently Shipyard is attempting to retrieve Tiller information to pass
to Armada. Once the dependent change is merged, Tiller will be moved
into the same pod as Armada, and this information will be retrieved
from the Armada configuration file. Therefore, Shipyard should no
longer be retrieving this information for Armada.

Co-Authored-By: Michael Beaver <>
Change-Id: I689e396ef9f184525f09f819f13c1490344481f2
2019-03-28 16:35:29 -05:00
Bryan Strassner 62cada246d Update dependencies on Armada, Drydock, Deckhand
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)

Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
2019-02-22 22:36:21 +00:00
Bryan Strassner 067db5f2a0 [FIX] Fix document build
Adds the SLUGIFY_USES_TEXT_UNIDECODE=yes option to the appropriate tox
jobs to allow document builds/regeneration of directories occur in
alignment with license requirements

Change-Id: Ie0d25183e6759d1ae48aa45c504479e1d0592d52
2019-02-22 15:43:08 -06:00
Bryan Strassner a11e962eef Move Airflow web container into Shipyard pod
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.

Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
2019-01-29 09:41:16 -06:00
Zuul 1474d7856f Merge "Raise specific errors during create configdocs" 2019-01-22 20:04:51 +00:00
Zuul ce8e8264a3 Merge "fix typos in API.rst &" 2019-01-22 20:01:11 +00:00
Zuul e232975db8 Merge "Remove those copy words occured twice times in total_garbage.yaml" 2019-01-04 18:28:16 +00:00
zhouxinyong 749b12b238 Remove those copy words occured twice times in total_garbage.yaml
Change-Id: I487529c1ad791c067d09e32023edf7eb3e3d40cf
2019-01-04 20:19:21 +08:00
lijunjie ecdb9ad345 Fix the misspelling of "except"
Change-Id: I85a40e078d2f5ec017d2aebedf79568de4163e44
2019-01-04 17:55:53 +08:00
Nishant Kumar 30950202b3 Raise specific errors during create configdocs
Currently shipyard raises same error for 2 scenarios-
    - Collection empty or resulted in no revision (Error code - 400)
    - Buffer is either not empty or the collection already
      exists in buffer (Error code - 409)

This PS enables shipyard to raise specific errors for the above
given scenarios which would equip consumers of Shipyard API client
to handle the exception gracefully.

Change-Id: I10860ca60f4fde4088cbb146283a2db305418320
2019-01-03 20:31:30 +00:00
Zuul c1b12b9a9e Merge "Updates cleartext-secrets RBAC Permissions" 2018-12-21 14:41:02 +00:00
Zuul 09d3238e78 Merge "Support clearing collections of configdocs" 2018-12-21 14:40:00 +00:00
melissaml 7b40eb190f fix typos in API.rst &
Change-Id: I5bb9f8e55580a926b2684aa2e62bb8cc139de3be
2018-12-12 01:15:48 +08:00
Bryan Strassner d6c72d19e6 Update to Airflow 1.10.1 and restore sysout
Updates to Airflow 1.10.1; See (1), (2) for some notes

Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.

A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)

Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.

Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.


Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
2018-12-05 12:37:14 -06:00
Bryan Strassner 667a538330 Support clearing collections of configdocs
Adds an option to create configdocs as an empty colleciton. This is done
as an explicit flag (--empty-collection) on the command as opposed to
using empty files to prevent accidental emtpy file loads leading to

Since this introduced a new flag value for the CLI, the CLIs using flag
values were updated to use the standard is_flag=True instead of the
flag_value=True or some other value when a boolean flag is expected.

Minor updates to CLI tests due to moving to responses 0.10.2


Change-Id: I489b0e1183335cbfbaa2014c1458a84dadf6bb0b
2018-11-30 13:36:14 -06:00
Aaron Sheffield 0cac1cbe2f Updates cleartext-secrets RBAC Permissions
- Adds an RBAC check when returning raw configdocs.

Change-Id: Ia4967ba4e1dfc49d44a3914cfa151177a49c3799
2018-11-28 10:01:40 -06:00
Zuul 03d7269b6a Merge "Ensure SY gets redacted rendered documents" 2018-11-14 22:34:23 +00:00
anthony.bellino 407cdc76d0 Ensure SY gets redacted rendered documents
- Introduced query parameter 'cleartext-secrets' yields
  cleartext secrets in encrypted documents if true, else
  the values are redacted.
- Introduced CLI flag '--cleartext-secrets' yields cleartext
  secrets in encrypted documents.

Change-Id: I8a950c8bded3550f4aab8e6dc662ea330c2fb73f
2018-11-14 18:32:37 +00:00
Bryan Strassner 84d99967bc Add configurable timeout for Drydock client
Adds configs to allow a drydock client to use a non-default read

Change-Id: Id4e4a235861165bfb5eb571684c8ce0be4181543
2018-11-13 13:10:07 -06:00
Zuul 7f3286b523 Merge "Expects Redacted Raw Documents" 2018-11-06 21:57:46 +00:00
lvxianguo bae15294c5 trivial: modify spelling error of response
Change-Id: Ie3d2ad641457852c15654fc6dd730f9d305e75fc
2018-11-01 17:15:49 +08:00
lvxianguo ad6445c3bf trivial: modify spelling error of test
Change-Id: I79d71d4596b3fd5ec625d5c2b749e8a9a0f7264e
2018-10-31 19:33:40 +08:00
Nishant Kumar a79733b4d8 Enable shipyard to use PBR
This PS enables shipyard to use PBR as rest
of the other OpenStack project.

Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.

Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
2018-10-25 14:23:50 +00:00
Aaron Sheffield d62f15123d Expects Redacted Raw Documents
- Adds a query parameter 'cleartext-secrets' to get full raw documents.
- Adds CLI flag to get full raw documents.

Change-Id: If38974c8433c8360cc47ae1273720ad76e87a6fd
2018-10-23 11:12:56 -05:00
j330k c90e017bea Making Python version uniform across all airship components.
Change-Id: I53febd3da36812e0bce0be5a80e57dca0c7804c3
2018-10-19 15:29:41 -05:00
Bryan Strassner f5774206e5 Add notes support for Builddata output
Enhances the workflow to include adding notes that contain the builddata
information associated with the Drydock steps. Part of adding this
support includes adding general notes support to all of the operators
that inherit from the UcpBaseOperator

Storyboard References:
Story: 2002797
Story: 2002796

Change-Id: I5e1a54d6373c4a523e2d4fe87796da4358f22055
2018-10-16 07:45:57 -05:00
Bryan Strassner 5a9abc73dd Modify note access methods
While iterating on the next steps of using notes, it became clear that
several changes to the output and access methods for notes needed
enhancements. This change introduces a new way to access a note's URL
information via a new API/CLI, while removing the resolution of URLs
from the existing note output. This supports the concept of "builddata"
coming back with sizes of 800kb or more - which really can never work
out inline in other data, especially in cases where there is
multiplicity of the information across many items.

New API: GET /notedetails/{id}
CLI: shipyard get notedetails/{id} and/or shipyard get notedetails {id}
Returns the resolution of the URL for a note, outputting the raw info as
the response (not structured in a JSON response).
The CLI will attempt to minimally format the response if it has inline
\n characters by replacing them will real newlines in the output (if the
output-format is set to either cli or format.  Raw format will be
returned as-is.

The existing notes responses are changed to not include the resolution
of the URL information inline, but rather provide the text:

  Details at notedetails/{id}

The CLI will interpret this and present:

  - Info available with 'describe notedetails/09876543210987654321098765'

This is an attempt to inform the user to access the note details that
way - luckily the API and CLI align on the term notedetails, as the word
details works well enough in the singular form presented by the CLI and
the plural form used by the API.
The ID returned is the unique id of the note (ULID format).

Notes that have no URL will return a 404 response from the API (and
an appropriately formatted value from the CLI).

This approach solves an issue beyond the large inline values from URLs;
providing a means to NOT resolve the URLs except in a one-at-a-time way.
Long lists of notes will no longer have the risk of long waits nor
needing of parallelization of retrieval of URLs for notes.

This change introduces an API-side sorting of notes by timestamp,
providing a chronological presentation of the information that may or
may not match the ULID or insertion ordering of the notes.

Additional feedback from peers about the output of noted indicated that
the CLI formatting of notes in general was in need of visual tuning. As
such, this change introduces changes to the formatting of the output
of notes from the CLI:

-  Notes for describing an item will be presented with a more specific
   header, e.g.: Action Notes: or Step Notes: instead of simply Notes.

-  Tables with notes will change the header from "Notes" to "Footnotes"
   give the user a better marker that the notes follow the current

-  Table footnotes will be presented in a table format similar to
   the following, with headings matching the kind of note being

   Step Footnotes    Note
   (1)               > blah blah blah
                     > yakkity yakkity
   (2)               > stuff stuff stuff stuff stuff stuff stuff
                     stuff stuff stuff
                       - Info available with 'describe notedetails/...
                     > things things things

Change-Id: I1680505d5c555b2293419179ade995b0e8484e6d
2018-10-16 07:45:02 -05:00
Bryan Strassner 06de84e0ab Add notes processing to the Shipyard API+CLI
Enhance the Shipyard API and CLI to retrieve notes that have been
specified against actions and steps. Includes a new reusable parameter
for verbosity.

Change-Id: I1c7f47c0346ce783dacd62b8bbc1fd35a0bf285b
2018-10-11 15:36:22 -05:00
Bryan Strassner 63a08c6a92 [FIX] mock notes helper and tests for node filter
Two testing only changes, one to mock the notes helper during the
action_api testing so it doesn't wait for network timeouts silently and
take a very long time to run, and secondly enhanced testing around the
production of node filters used with deployment groups to validate that
rack_names are being handled in the desired way when producing node
filters for Drydock.

Change-Id: I5439e82333f40e91c270fa52c466731b4bbf1f2f
2018-10-10 22:41:05 -05:00
Zuul 282c5699b9 Merge "Add notes common code for Shipyard" 2018-10-09 20:24:13 +00:00
skovaleff d6b7506869 Fix get_pod_logs logic to pick up accurate pods
Shipyard/Airflow Worker selection of pods for log retrieval inaccurate

Change-Id: I8aade93db6e657da220254242bc51a166853d965
2018-10-09 08:05:00 -07:00
skovaleff 124de57262 Fix typo in a description
Change-Id: I4f3892af2a4e7cea97157754cc7ad522043f00f6
2018-10-05 16:10:12 -07:00
Zuul 0593989f00 Merge "Add Shipyard profiler" 2018-10-05 21:20:26 +00:00
Zuul 3986faaf8b Merge "Refactor validations retrieval for performance" 2018-10-05 21:20:26 +00:00
Bryan Strassner 35d6089d32 Add notes common code for Shipyard
Adds the common/shared code to support notes in Shipyard such that this
component can be reused between the Shipyard API and the workflows.
Includes a single example of creating and retreiving notes as part of
action creation, that will likely be changed in a future change.

Note that this change doesn't include changes to the CLI to represent
notes outwardly, but the first example can be seen using the
--output-format=format option (shipyard get actions)

Change-Id: I2f87713eb74dae312912ff4c36e6ae30a569ea38
2018-10-05 15:40:48 -05:00
Zuul 4ecf2015bd Merge "Fix typo in description" 2018-10-05 17:48:46 +00:00
Zuul af1a3dd900 Merge "actions: Add Shipyard action to test site" 2018-10-05 15:46:40 +00:00