Commit Graph

769 Commits (a0da68409a1224ce84e4c7aa6487e0a5a9c681ee)

Author SHA1 Message Date
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.


Signed-off-by: Sean Eagan <>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Sean Eagan 27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
anthony.bellino 3a27007e8a (zuul) Fix Shipyard Post Gates
Change-Id: I9f6f633399069f28c47c381faf4230e4f22276e2
2021-09-10 13:13:38 -07:00
Maximilian Weiss 88286bc690 Gate fixes for Shipyard
* Fixed typo for jsonschema version

* Setuptools version forced to <58 to allow usage of use_2to3, which
  some dependencies require

* Fixed scripting error that allowed Airflow installation to silently

Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
2021-09-09 20:23:05 +00:00
Maximilian Weiss ac236ef52a Update HTK stable commit to 0.2.19
Update helm-toolkit stable commit to merge of this change:

Change-Id: I07e82486c901b558ab2f9619d5663593305c504f
2021-08-17 17:50:35 +00:00
Maximilian Weiss 99e253092a Update helm installation script
Updates the helm installation script to download and install v2.17.0
from (instead of v2.16.9 from

Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
2021-08-17 17:50:35 +00:00
anthony.bellino 00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Rick Bartra 35e5a7f796 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
2020-09-30 01:19:51 +00:00
Zuul fb1c281b83 Merge "Accelerate YAML operations with LibYAML" 2020-09-28 17:12:04 +00:00
Zuul 24d301851a Merge "Include LibYAML in container builds" 2020-09-28 17:01:55 +00:00
Zuul 19a11a7766 Merge "Fix airflow quicktest error" 2020-09-28 17:01:54 +00:00
Andrii Ostapenko af63252e8d
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5
2020-09-24 19:43:13 -05:00
Phil Sphicas 4e7ff997a6 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
2020-09-24 05:09:46 +00:00
Phil Sphicas 6b5f9c4db4 Include LibYAML in container builds
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.

Change-Id: I6ebafa1eb2af8841b86b64fd119dcb581e6c18a7
2020-09-24 04:35:26 +00:00
Phil Sphicas 86d2e0d9f2 Fix airflow quicktest error
airflow quicktest is throwing an error when trying to start the
    can't open file '_cmd': [Errno 2] No such file or directory

Change-Id: I3fddc1cdd3daeacc09a5924287d711a72878c5e3
2020-09-24 04:35:26 +00:00
Rick Bartra b117fadbdb fix: Docker image build jobs
Update the deb-docker path to fix the docker image build jobs

Change-Id: I31660fcc5358fc67caba39fdaf38b60b13e7eaff
2020-09-22 09:23:18 -05:00
KAVVA, JAGAN MOHAN REDDY (jk330k) 796f2ddcca Move Tiller version to 2.16.9
Update Helm chart for Shipyard to use Tiller version 2.16.9.

Change-Id: Ia61098db57c741f36864db084d39dd9b27d25251
2020-09-16 17:00:54 +00:00
Zuul 8b69cebeea Merge "fix: Gate fix for Shipyard docs and pep8 gates" 2020-09-15 19:52:36 +00:00
Rick Bartra d38078cce3 fix: Gate fix for Shipyard docs and pep8 gates
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.

Additionally, address pep8 violations that arise from using a newer
version of flake8

Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
2020-09-15 18:16:45 +00:00
Bartra, Rick (rb560u) 1ae9a7c4be Improve logging of Shipyard _execute_task method during failure or time out
In the case where a Drydock task either fails or times out, it is not very
clear which Drydock task is being referred to. This commit updates the log
messages to include the Drydock task-id that has either failed or timed out.
Additionally, debug logging is added to log the Drydock task status. The reason
for this is because there are situations where a Drydock task is still running,
but on the Shipyard side the task has timed out and therefore, Shipyard shows
the task (i.e. prepare_nodes or deploy_nodes) as failed. It is good to get a point
in time reference to the task state for a couple of reasons:
1) To verify via logging that Shipyard is reporting Drydock tasks (success,
   failures, partial_successes, etc) accurately
2) Later on if Drydock shows a node as deployed, then the task can be
   queried and the state can be checked to see if the task was indeed a success.

Change-Id: I7050338e2f92ad548e639e2ea4059a520c27e686
2020-08-28 14:00:13 -04:00
Mahmoudi, Ahmad (am495p) a5e57879ab Override uwsgi default config
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
  improvements since 2018.

Upgraded uwsgi to bring in bug fixes since 2018.

For background information for this change please see:

Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
2020-08-06 02:14:07 +00:00
KHIYANI, RAHUL (rk0850) 841d0ee2c8 Add the missing readOnly-fs flag for airflow-web containers
Change-Id: Ic31288d2f88c9610621ef1b74d8de813ea985ca8
2020-07-17 16:00:26 +00:00
KHIYANI, RAHUL (rk0850) db37122336 Implement helm-toolkit snippet to airflow pods/containers
This updates the airflow chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag

Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
2020-07-09 20:36:39 +00:00
Zuul 66d410779c Merge "Add configmap-hash annotations for Shipyard & Airflow" 2020-07-06 21:48:15 +00:00
DODDA, PRATEEK REDDY 5247fed4ba Add configmap-hash annotations for Shipyard & Airflow
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.

These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.

Change-Id: I59eb516086c4fd41f7c18923f86f135101656af8
2020-07-06 12:09:18 -05:00
KHIYANI, RAHUL (rk0850) 02929cfc44 Implement helm-toolkit snippet to shipyard pods/containers
This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag to true

Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
2020-07-02 09:25:11 -05:00
Zuul f6e4066a83 Merge "Enabling Apparmor profile to shipyard init containers" 2020-06-26 19:27:14 +00:00
Zuul 83501c83c2 Merge "Uplift celery to address memory leak" 2020-06-26 16:46:54 +00:00
Zuul 1599e49e70 Merge "Update Airship vulnerability link" 2020-06-26 15:08:25 +00:00
DODDA, PRATEEK 9831e545c9 Enabling Apparmor profile to shipyard init containers
Remove OSH Authors copyright

The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
2020-06-26 09:11:41 -05:00
Ahmad Mahmoudi e2a3d13875 Uplift celery to address memory leak
Uplifted celery to 4.4.2 to address memory leak issue raised
This issue is addressed in celery 4.4.2

Change-Id: I26c403513ba48308044d69130d33561b314fd209
2020-06-25 21:04:22 +00:00
Mahmoudi, Ahmad (am495p) c13c9940dc (fix) Added task to install pip3
Added task to install pip3 addres the zuul issue not finding pip.
The zuul job airship-shipyard-airskiff-deployment failes with:
sudo -H -E pip install 'cmd2<=08.7'
sudo: pip: command not found

Change-Id: Ib43aced2c054ecc38b2ce61377b735543c0a9f9f
2020-06-24 16:06:44 +00:00
Ahmad Mahmoudi fb9cc9a65e (fix) - updated to pip3
Updated to pip to pip3 to address zuul gate issue.
Set the ensure_global_symlinks to true for zuul-jobs to set
symlink for tox path."

Change-Id: I1b0634ef18328bccc4d6929072f53db779d70ef1
2020-06-18 19:51:47 +00:00
DODDA, PRATEEK REDDY (pd2839) 0957ede4a5 (fix) Update amqp
Locked amgp to 2.6.0 as per the latest release on 06/01/2020.

Change-Id: I2edc5eb1959d71454236eaf14560fd38c4d1140b
2020-06-08 18:21:10 +00:00
Drew Walters b0012a26e4 Update Airship vulnerability link
The Airship vulnerability documentation has moved [0]. This change
updates to point to the correct location.


Change-Id: Iea843a3399bc7836f5645c3ca81603e2e9ca7356
Signed-off-by: Drew Walters <>
2020-05-18 21:27:25 +00:00
Prateek Dodda e066274b8b Implement Security Context for Airflow_Scheduler
This adds the container security context to set readOnlyRootFilesystem
to true

Change-Id: Ia9cad50decfcf9638e8fc1cf5d652ee72d978a40
2020-04-24 17:03:24 -05:00
Phil Sphicas c8c8934c77 Armada DAG: increase retries from 3 to 5
The current Armada DAG allows for 3 retries, reduced from 10 here:

This is sometimes insufficient, especially in cases where chart updates
of underlying Airship or Kubernetes components are upgraded and pods are
restarted underneath the airflow workers. The chart installation may be
successful, but an Armada retry may still be consumed.

This change increases the number of retries to 5. This will allow Armada
to progress further through the manifest if there is a disruption after
a chart is successfully installed. The tradeoff is that Armada may try
to repeatedly install a chart that keeps failing in the same way,
delaying the ultimate failure of the deployment.

Change-Id: I1fad7b1d95af061595680a76d24c6d323b365a67
2020-04-23 20:33:58 +00:00
Ahmad Mahmoudi 0091670583 (fix) Updated WTForms and cleaned psycopg2
1. Locked the WTForms to 2.2.1 to address the import issue with
   wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
   April 21/2020. This release causes shipyard gate fail with
   import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
   apache-airflow extra package postgres, and resoted the newer
   release psycopg2-binary==2.8.4, to be used instead.

Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
2020-04-22 17:06:08 +00:00
Zuul ec46396487 Merge "Upgrade apache-airflow to 1.10.5" 2020-04-02 15:50:09 +00:00
Zuul 8b6bd94410 Merge "Implement Security Context for Airflow_Worker" 2020-04-02 13:32:14 +00:00
Prateek Dodda cc0bfac0c2 Implement Security Context for Airflow_Worker
This adds the container security context to set
readOnlyRootFilesystem to true

Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
2020-03-31 14:14:03 +00:00
Ahmad Mahmoudi df751e302c Upgrade apache-airflow to 1.10.5
Upgraded apache-airflow to 1.10.5 to address the database migration issue

Change-Id: Ib8678cddb2deb4f5ef944d026e0d2633a326d9cc
2020-03-30 05:21:31 +00:00
Drew Walters 5b1af08d32 Add
All Airship projects are moving to GitHub issues. This change adds a
GitHub security policy that links to the official Airship vulnerability
management process [0]. When users on GitHub click "New Issue" on this
GitHub repository, they will see an option to report a security
vulnerability, which will direct them to our official policy.


Change-Id: Iaf060dd0085c21f0c4f18f100e3e053b5ceedbed
Signed-off-by: Drew Walters <>
2020-03-26 21:03:00 +00:00
KHIYANI, RAHUL (rk0850) 29824c78b4 [Fix] Fixing shipyard endpoints path name
fixing path helm-toolkit to shipyard


Change-Id: I73bf30e0c27365802b730d27e6ecfd28092de24e
2020-03-18 23:08:33 -05:00
Ahmad Mahmoudi 70410cc478 (fix) Address image build issues, bionic
- With bionic image based shipyard docker images, uwsgi crashes
  with segmentation fault, when it tries to load the psycopg2 library,
  causing the api become unreachable on both shipyard docker images.
  This happens because psycopg2 2.7.x and uwsgi binary wheels are built
  with incompatible ssl libraries. This patch upgrades psycopg2 to the
  latest release to address this issue.

- The existing image build script cannot run in a docker or a pod,
  based pipeline because of two reasons:
  - The build script runs a docker (docker-in-docker) and mounts a
    In a dind case, volume bind mounts will not work, because the nested
    container will need the host file system's path for the source path.
  - The shipyard service listens to its exposed service port in the
    nested docker network namespace, which is not reachable from the host
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.

Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
2020-03-10 03:23:05 +00:00
Zuul 30f3a989c7 Merge "Adding default apparmor profile to shipyard components" 2020-02-19 14:28:54 +00:00
NarlaSandeepNarlaSaibaba 5aa0cde5f0 Adding default apparmor profile to shipyard components
Change-Id: Idfc103c85bc95c8cd0a48aa0c18a17a4b1d12d3f
2020-02-17 09:52:37 -06:00
Drew Walters dfa51979c2 CI: Fix airskiff node name
The airskiff job is looking for a node named "Ubuntu-Bionic", but the
only node available in its nodeset is "primary". This change updates the
playbook to use the proper name.

Change-Id: Ib3a8de0918c7a9e5abb7fb71a20ae261f68b3259
Signed-off-by: Drew Walters <>
2020-02-12 18:10:43 +00:00
Ahmad Mahmoudi 3d987309b8 Locked Werkzeug package release to 0.16.1
The latest Werkzeug package release 1.0.0, released on February 8,
is not backwards compatible with the earler releases of this package,
which is used in Flask, used by apache airflow.

This causes shipyard make image job fail, with missing import errors
from the Werkzeug library.

This change locks the Werkzeuz package release to the last compatible
release with the apache airflow in shipyard.

Change-Id: I54dad4ccc1858f4d5986c6e8e9fbf8f5d9847158
2020-02-12 01:23:15 +00:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00