Follow up change for deployment-status ConfigMap [0] adding a few
things that were either missed or need to be added, especially unit
tests for the new DeploymentStatusOperator class
[0] https://review.opendev.org/#/c/660197/
Change-Id: Ieaf55dc7725ea92062fde7997a676726b2396253
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps
fail.
This operator has been added to the deploy_site, update_site, and
update_software DAGs.
Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
This change adds support in the concurrency_check plugin to push its
status to xcom so other components can easily check if it has passed.
This is especially useful for components that need to always run, but
still want to respect the concurrency check.
This is also a slight refactor of XcomPuller to make it import its dag
names from dag_names.py
Change-Id: I9ca6b43d7789d9499121384d4427835e296c44b8
This patch set adds functionality to Shipyard's action_helper module
to generate a set of "deployment-status" data. This data can be used
to denote the status and result of a Shipyard action and will be
utilized in a future feature to help track actions/deployments.
This new functionality is not yet being used, and actual Shipyard
functionality should not be altered with this patch set.
Change-Id: I838abfd2144f08708b3fc5a53cc76f124aedff04
In all honesty, the warnings that were output about unsafe yaml.load
while running unit tests in Shipyard were just kind of annoying.
They always made me think there was a unit test failure for a split
second. This patch set updates yaml.load to yaml.safe_load in the
configdocs helper unit tests. The unit tests themselves are
effectively unchanged, but now there are no warnings returned.
Change-Id: I65a5d55fbe9eb690ad534a5b7df882762bf5a9fb
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.
Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.
Change-Id: I754617de81f628a24232e890b12b157ba6731c25
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard
Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.
Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.
Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.
[0] https://review.openstack.org/633873
[1] https://review.openstack.org/634068
[2] https://review.openstack.org/638115
Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Armada API client. The Armada client was updated to handle the
additional parameter [1]; however, the Shipyard version Armada uses was
not updated. This change uplifts the version of Armada in
requirements.txt for the Airflow component of Shipyard to a version that
supports the aforementioned, additional, user-context parameter.
[0] https://review.openstack.org/633873
[1] https://review.openstack.org/638124
Change-Id: I7dfbda842eb136858fbb6d891b96d057a977c913
This PS adds entry in log for user id and passes on the context
maker to other Airship components from Shipyard during API call.
This will ensure easy tracing of user and context through log
tracing.
Change-Id: Ib9bfa8f20b641f8bb6c2dca967d9388e30d5735c
Currently Shipyard is attempting to retrieve Tiller information to pass
to Armada. Once the dependent change is merged, Tiller will be moved
into the same pod as Armada, and this information will be retrieved
from the Armada configuration file. Therefore, Shipyard should no
longer be retrieving this information for Armada.
Co-Authored-By: Michael Beaver <michaelbeaver64@gmail.com>
Depends-On: https://review.openstack.org/#/c/632788/
Change-Id: I689e396ef9f184525f09f819f13c1490344481f2
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)
Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
Adds the SLUGIFY_USES_TEXT_UNIDECODE=yes option to the appropriate tox
jobs to allow document builds/regeneration of directories occur in
alignment with license requirements
Change-Id: Ie0d25183e6759d1ae48aa45c504479e1d0592d52
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.
Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
Currently shipyard raises same error for 2 scenarios-
- Collection empty or resulted in no revision (Error code - 400)
- Buffer is either not empty or the collection already
exists in buffer (Error code - 409)
This PS enables shipyard to raise specific errors for the above
given scenarios which would equip consumers of Shipyard API client
to handle the exception gracefully.
Change-Id: I10860ca60f4fde4088cbb146283a2db305418320
Updates to Airflow 1.10.1; See (1), (2) for some notes
Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.
A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)
Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.
Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.
(1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
(2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1
Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
Adds an option to create configdocs as an empty colleciton. This is done
as an explicit flag (--empty-collection) on the command as opposed to
using empty files to prevent accidental emtpy file loads leading to
frustration.
Since this introduced a new flag value for the CLI, the CLIs using flag
values were updated to use the standard is_flag=True instead of the
flag_value=True or some other value when a boolean flag is expected.
Minor updates to CLI tests due to moving to responses 0.10.2
Depends-On: https://review.openstack.org/#/c/614421/
Change-Id: I489b0e1183335cbfbaa2014c1458a84dadf6bb0b
This PS enables shipyard to use PBR as rest
of the other OpenStack project.
Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.
Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
- Adds a query parameter 'cleartext-secrets' to get full raw documents.
- Adds CLI flag to get full raw documents.
Change-Id: If38974c8433c8360cc47ae1273720ad76e87a6fd
Enhances the workflow to include adding notes that contain the builddata
information associated with the Drydock steps. Part of adding this
support includes adding general notes support to all of the operators
that inherit from the UcpBaseOperator
Storyboard References:
Story: 2002797
Story: 2002796
Change-Id: I5e1a54d6373c4a523e2d4fe87796da4358f22055
While iterating on the next steps of using notes, it became clear that
several changes to the output and access methods for notes needed
enhancements. This change introduces a new way to access a note's URL
information via a new API/CLI, while removing the resolution of URLs
from the existing note output. This supports the concept of "builddata"
coming back with sizes of 800kb or more - which really can never work
out inline in other data, especially in cases where there is
multiplicity of the information across many items.
New API: GET /notedetails/{id}
CLI: shipyard get notedetails/{id} and/or shipyard get notedetails {id}
Returns the resolution of the URL for a note, outputting the raw info as
the response (not structured in a JSON response).
The CLI will attempt to minimally format the response if it has inline
\n characters by replacing them will real newlines in the output (if the
output-format is set to either cli or format. Raw format will be
returned as-is.
The existing notes responses are changed to not include the resolution
of the URL information inline, but rather provide the text:
Details at notedetails/{id}
The CLI will interpret this and present:
- Info available with 'describe notedetails/09876543210987654321098765'
This is an attempt to inform the user to access the note details that
way - luckily the API and CLI align on the term notedetails, as the word
details works well enough in the singular form presented by the CLI and
the plural form used by the API.
The ID returned is the unique id of the note (ULID format).
Notes that have no URL will return a 404 response from the API (and
an appropriately formatted value from the CLI).
This approach solves an issue beyond the large inline values from URLs;
providing a means to NOT resolve the URLs except in a one-at-a-time way.
Long lists of notes will no longer have the risk of long waits nor
needing of parallelization of retrieval of URLs for notes.
This change introduces an API-side sorting of notes by timestamp,
providing a chronological presentation of the information that may or
may not match the ULID or insertion ordering of the notes.
Additional feedback from peers about the output of noted indicated that
the CLI formatting of notes in general was in need of visual tuning. As
such, this change introduces changes to the formatting of the output
of notes from the CLI:
- Notes for describing an item will be presented with a more specific
header, e.g.: Action Notes: or Step Notes: instead of simply Notes.
- Tables with notes will change the header from "Notes" to "Footnotes"
give the user a better marker that the notes follow the current
table.
- Table footnotes will be presented in a table format similar to
the following, with headings matching the kind of note being
produced.
Step Footnotes Note
(1) > blah blah blah
> yakkity yakkity
(2) > stuff stuff stuff stuff stuff stuff stuff
stuff stuff stuff
- Info available with 'describe notedetails/...
> things things things
Change-Id: I1680505d5c555b2293419179ade995b0e8484e6d
Enhance the Shipyard API and CLI to retrieve notes that have been
specified against actions and steps. Includes a new reusable parameter
for verbosity.
Change-Id: I1c7f47c0346ce783dacd62b8bbc1fd35a0bf285b
Two testing only changes, one to mock the notes helper during the
action_api testing so it doesn't wait for network timeouts silently and
take a very long time to run, and secondly enhanced testing around the
production of node filters used with deployment groups to validate that
rack_names are being handled in the desired way when producing node
filters for Drydock.
Change-Id: I5439e82333f40e91c270fa52c466731b4bbf1f2f
Adds the common/shared code to support notes in Shipyard such that this
component can be reused between the Shipyard API and the workflows.
Includes a single example of creating and retreiving notes as part of
action creation, that will likely be changed in a future change.
Note that this change doesn't include changes to the CLI to represent
notes outwardly, but the first example can be seen using the
--output-format=format option (shipyard get actions)
Change-Id: I2f87713eb74dae312912ff4c36e6ae30a569ea38
Michael Beaver