Commit Graph

667 Commits (02533cb08594bb4add62dc10fa6c22a51b1fe75c)

Author SHA1 Message Date
Michael Beaver 02533cb085 Add unit tests and fix comments/samples
Follow up change for deployment-status ConfigMap [0] adding a few
things that were either missed or need to be added, especially unit
tests for the new DeploymentStatusOperator class


Change-Id: Ieaf55dc7725ea92062fde7997a676726b2396253
2019-05-29 15:02:59 -05:00
Michael Beaver 53e863954b Store status of deployment in a ConfigMap
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps

This operator has been added to the deploy_site, update_site, and
update_software DAGs.

Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
2019-05-28 08:32:07 -05:00
Michael Beaver b5469c39ec Create Xcom Pusher/Puller for concurrency_check
This change adds support in the concurrency_check plugin to push its
status to xcom so other components can easily check if it has passed.
This is especially useful for components that need to always run, but
still want to respect the concurrency check.

This is also a slight refactor of XcomPuller to make it import its dag
names from

Change-Id: I9ca6b43d7789d9499121384d4427835e296c44b8
2019-05-25 19:02:29 -05:00
Carter, Matthew (mc981n) c19580378b Add deployment status generation to Shipyard
This patch set adds functionality to Shipyard's action_helper module
to generate a set of "deployment-status" data. This data can be used
to denote the status and result of a Shipyard action and will be
utilized in a future feature to help track actions/deployments.

This new functionality is not yet being used, and actual Shipyard
functionality should not be altered with this patch set.

Change-Id: I838abfd2144f08708b3fc5a53cc76f124aedff04
2019-05-25 19:02:29 -05:00
Kaspars Skels a9181dcc60 Encrypt git mirroring ssh_key to specific project
Change-Id: Icc1e3077a9e0efcfd53e92eab325fdfe264abb65
2019-05-23 13:08:56 -05:00
Zuul 8922aabf38 Merge "Add Zuul job for mirroring to GitHub" 2019-05-23 09:54:06 +00:00
Kaspars Skels 3bce3c41ca Add Zuul job for mirroring to GitHub
Change-Id: I318020a92d0d81fa8fb206ee9e7366cc8982255e
2019-05-22 16:21:20 -05:00
Sean Eagan 912821f22a Move to helm 2.14.0
Change-Id: I6e0c2a849965ac0c8123d72a3be4470dfcaf451f
2019-05-21 13:49:27 -05:00
Andreas Jaeger 8bfa323814 Fix rtd publishing
The trigger URL is [0].

The tooling sets rtd_project_name to "airmada" but the project is
named "airship-xxxx" in RTD. Introduce rtd_project_name and set it to
correct value.

See also [1].

[0]{{ rtd_project_name }}/{{ rtd_webhook_id }}

Change-Id: I57ab07ac8b5cf891437b41cea74b09640df026be
2019-05-17 16:04:43 +02:00
Arun Kant cd6f154da5 Adding opensuse support in image building of airflow and shipyard
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.

Added new document section for multiple distro support.

Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
2019-05-14 09:52:24 -07:00
Zuul d479d6b969 Merge "Validate existence of "deployment-version" during create configdocs" 2019-05-13 15:39:38 +00:00
Zuul 38c6faab7f Merge "Validate existence of "deployment-version" doc on configdocs commit" 2019-05-13 15:37:02 +00:00
Carter, Matthew (mc981n) c25a78fef4 Update yaml.load to yaml.safe_load in Shipyard unit tests
In all honesty, the warnings that were output about unsafe yaml.load
while running unit tests in Shipyard were just kind of annoying.
They always made me think there was a unit test failure for a split
second. This patch set updates yaml.load to yaml.safe_load in the
configdocs helper unit tests. The unit tests themselves are
effectively unchanged, but now there are no warnings returned.

Change-Id: I65a5d55fbe9eb690ad534a5b7df882762bf5a9fb
2019-05-09 10:34:52 -05:00
Carter, Matthew (mc981n) 0761099337 Validate existence of "deployment-version" during create configdocs
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.

Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.

Change-Id: I754617de81f628a24232e890b12b157ba6731c25
2019-05-09 08:33:52 -05:00
Carter, Matthew (mc981n) 64171aacf4 Validate existence of "deployment-version" doc on configdocs commit
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
2019-05-09 08:25:37 -05:00
Zuul e2a87a1830 Merge "Be configuration driven when referencing document names/schemas" 2019-05-08 18:20:59 +00:00
Carter, Matthew (mc981n) 222b074cb1 Be configuration driven when referencing document names/schemas
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.

Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
2019-05-02 10:47:38 -05:00
Zuul ae1ab1014e Merge "tools: add test_release script" 2019-05-01 22:41:57 +00:00
Zuul 1d519520c8 Merge "Don't log OS_PASSWORD in auth.log" 2019-04-29 18:44:37 +00:00
Zuul 902ee1b7af Merge "Update ha_celery policy pattern" 2019-04-29 18:14:19 +00:00
Crank, Daniel (dc6350) 0ab1bf552d Don't log OS_PASSWORD in auth.log
By default, all sudo commands are logged to auth.log with their
full command line. Previously, Shipyard scripts called 'sudo docker'
with -e OS_PASSWORD=foo, resulting in the password value appearing
in auth.log in plaintext. This change adds -E to the sudo command
to pass the user's environment through, and removes the value from
-e OS_PASSWORD which tells Docker to use the environment value
directly. This prevents the password value from being logged.

Change-Id: Ifcf7f6525876144a609ff42be42da57a3f7f6f60
2019-04-26 17:17:06 -05:00
Zuul 8d0b3f671f Merge "CI: Add Airskiff check" 2019-04-26 15:03:50 +00:00
Zuul f395a2efb9 Merge "Return error if execution of the step fails" 2019-04-24 17:48:36 +00:00
anthony.bellino d294a1939a Update ha_celery policy pattern
Updating pattern to include the default queue and all celery matching queues.

Change-Id: Id85951e9d84c2e0d43a06b3509ff364f71110092
2019-04-22 18:11:47 +00:00
Drew Walters 5556227634 CI: Add Airskiff check
This commit adds a non-voting Zuul job that deploys Memcached using the
Airship-Treasuremap airskiff site and submitted Shipyard changes.


Change-Id: Ibe9c6c94406bcdac4c39493b694535baf8ad2719
2019-04-22 16:39:58 +00:00
OpenDev Sysadmins 46875d8ac4 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at with any
questions you may have.
2019-04-19 19:52:20 +00:00
Crank, Daniel (dc6350) 70328e267b Airflow: Update Deckhand / Drydock version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.


Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
2019-04-18 11:50:41 -05:00
Bryan Strassner b575df40e6 Update mq queue mirror directive to match vhost
Updates the definition for queue mirroring to use the same vhost as the
queue itself is defined to be.

Change-Id: Ibb631d7adb237fa17c5e853a9e9e35297a525782
2019-04-17 16:30:14 -05:00
Zuul aa909ed2eb Merge "CI: Add chart build job for latest Helm toolkit" 2019-04-17 16:28:31 +00:00
Bryan Strassner 6f995d1fb2 Update script
The current script is fragile to the introduction of a DEBUG logging
level. This change addresses the specifics of logging records being
returned when invoking the airflow command enough to make the dag_status
result end up being the last result as the script desires, so the script
is returned to working order, even if the logging level is set to DEBUG.

This change goes a little further to the point of removing all records
that indicate a logging level of DEBUG|INFO|WARN|ERROR.

Change-Id: Ic72358097e3a476e20fa3713008d0d0bcd35463e
2019-04-17 08:36:35 -05:00
Evgeny L 4a065c3270 Return error if execution of the step fails
When Shipyard fails to execute some of the steps, the status
of the action is always "Complete", determine the success
of the execution based not only on action status, but
also on the status of every step within the action.

Change-Id: If7f71b55c2aed0322edef9c811ed1906400b0913
2019-04-16 21:48:07 +00:00
Zuul b36bf8abb7 Merge "Shipyard: Add pod/container security context - deployment-shipyard" 2019-04-16 15:49:57 +00:00
Drew Walters 962126ed4c airflow: Update image reqs to match src reqs
The airflow image has a requirements file that is separate from the one
located in src/bin/shipyard_airflow, which was missed in a previous
uplift patch [0]. This change updates all versions in the image
requirements file to match the source code requirements.


Change-Id: I238bba4038e5cd8364711c2c5d1f78f51fc76948
2019-04-15 19:25:28 +00:00
Zuul ad93fc1b02 Merge "images: Update default image tag to `latest`" 2019-04-15 19:01:49 +00:00
Drew Walters cdad9e7cbc airflow: Update Armada version
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Armada API client. The Armada client was updated to handle the
additional parameter [1]; however, the Shipyard version Armada uses was
not updated. This change uplifts the version of Armada in
requirements.txt for the Airflow component of Shipyard to a version that
supports the aforementioned, additional, user-context parameter.


Change-Id: I7dfbda842eb136858fbb6d891b96d057a977c913
2019-04-15 12:14:45 -05:00
Smruti Soumitra Khuntia 9c5270b616 User context tracing through logging
This PS adds entry in log for user id and passes on the context
maker to other Airship components from Shipyard during API call.

This will ensure easy tracing of user and context through log

Change-Id: Ib9bfa8f20b641f8bb6c2dca967d9388e30d5735c
2019-04-04 13:19:02 +00:00
Scott Hussey 7ff21610a5 (zuul) Fix image publish job
- Fix issue in post pipeline image publish job introduced
  by Ansible update

Change-Id: Ia97f2927980f5ee5c2d5adf23a5a016b8d3b1c9f
2019-04-03 15:23:27 -05:00
Drew Walters 95179f53ff CI: Add chart build job for latest Helm toolkit
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra. This job
should serve as an indicator of when it's safe to advance the version of
Helm toolkit used by Airship.

Additionally, this commit modifies all Helm chart lint jobs to run on
each commit, regardless of the files modified by a change. This should
not introduce a noticeable difference in CI runtime, as these jobs
execute quicker than the tox jobs.

Change-Id: Ife3b285a2f9a160df9724e9dcb230243d06be3ab
2019-04-03 14:26:37 +00:00
Zuul 279ca0f700 Merge "Adds HTTP Security Headers" 2019-04-02 18:33:24 +00:00
Rahul Khiyani 25defd8ca7 Shipyard: Add pod/container security context
- deployment-shipyard

This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
allowPrivilegeEscalation to false and readOnlyRootFilesystem to true

Change-Id: Idb1b848847eaec2b6e24389c063b7ece2973c4dc
2019-04-02 15:37:59 +00:00
Zuul 8cfc2b228d Merge "Remove Shipyard queries for Tiller information" 2019-04-01 18:17:05 +00:00
Nishant kumar afd2788729 Remove Shipyard queries for Tiller information
Currently Shipyard is attempting to retrieve Tiller information to pass
to Armada. Once the dependent change is merged, Tiller will be moved
into the same pod as Armada, and this information will be retrieved
from the Armada configuration file. Therefore, Shipyard should no
longer be retrieving this information for Armada.

Co-Authored-By: Michael Beaver <>
Change-Id: I689e396ef9f184525f09f819f13c1490344481f2
2019-03-28 16:35:29 -05:00
Drew Walters fc16d9c2e3 tools: Update Helm to v2.13.1
Helm v2.13.1 has been released [0]. This change uplifts Helm to v2.13.1
to match other Airship projects and pins the Shipyard chart linting gate
to the latest version of Helm toolkit.


Change-Id: Id9b15ad5f3253332ef0f8129b62a7b7510ab475c
2019-03-28 15:25:17 +00:00
Drew Walters 3af62864d6 images: Update default image tag to `latest`
This commit changes the default image tag for Shipyard and Airflow
images from `untagged` to `latest` to match other Airship projects.

Change-Id: Id1f9d56784e51f0631280188ba1b898fde8f56ea
2019-03-22 18:42:48 +00:00
Aaron Sheffield 12de088b9d Updating Docker Gate use of zuul.newrev
- Zuul updated ansible to 2.7, no longer allows missing variables.
- Using default value when it isn't available.

Based on Aaron Sheffield's PS for Pegleg:

Change-Id: I02495bc793021b429e0be62ecac0ed45e930484c
2019-03-22 12:53:55 -05:00
Dustin Specker d5c53780de tools: add test_release script
For now, this script requires specifying a Helm release to test.

Shipyard does not actually require a Helm release. If a Helm release is
not passed as param to test_site, Shipyard will run all Helm tests. This
script does not support this scenario. This script only supports testing
a single Helm release.

Change-Id: I67eb74d7d97d6460872eb1495194c41f08bae055
2019-03-15 12:56:27 -05:00
Zuul 041cbcbafe Merge "tools: generalize execute_shipyard_action" 2019-03-12 19:59:35 +00:00
Zuul 38956c7b27 Merge "[Database] Shipyard DB changes" 2019-03-07 21:21:39 +00:00
Nishant Kumar d9f145e2a6 [Database] Shipyard DB changes
- Use helm-toolkit for DB initialization [0]
- Create DB auxiliary Job for shipyard specific
  additonal DB operations
- Refactor Job dependencies



Change-Id: I093671f9bce747b491f22dd8f38f597bd9dae9af
2019-03-07 15:16:39 +00:00
Dustin Specker c234b72abd tools: generalize execute_shipyard_action
This enables passing any combination of actions and parameters to this
script.  This also makes it easier to add scripts for additional actions
by not having to modify execute_shipyard_action.

Updated redeploy_server to pass in servers as a param.

Change-Id: I590183c28e8c66997ab85470902dbe9576c5c5f6
2019-02-28 15:16:06 -06:00