The current Armada DAG allows for 3 retries, reduced from 10 here:
bef8eecac1
This is sometimes insufficient, especially in cases where chart updates
of underlying Airship or Kubernetes components are upgraded and pods are
restarted underneath the airflow workers. The chart installation may be
successful, but an Armada retry may still be consumed.
This change increases the number of retries to 5. This will allow Armada
to progress further through the manifest if there is a disruption after
a chart is successfully installed. The tradeoff is that Armada may try
to repeatedly install a chart that keeps failing in the same way,
delaying the ultimate failure of the deployment.
Change-Id: I1fad7b1d95af061595680a76d24c6d323b365a67
1. Locked the WTForms to 2.2.1 to address the import issue with
wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
April 21/2020. This release causes shipyard gate fail with
import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
apache-airflow extra package postgres, and resoted the newer
release psycopg2-binary==2.8.4, to be used instead.
Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
This adds the container security context to set
readOnlyRootFilesystem to true
Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
All Airship projects are moving to GitHub issues. This change adds a
GitHub security policy that links to the official Airship vulnerability
management process [0]. When users on GitHub click "New Issue" on this
GitHub repository, they will see an option to report a security
vulnerability, which will direct them to our official policy.
[0] https://airship-docs.readthedocs.io/en/latest/security/vulnerabilities.html
Change-Id: Iaf060dd0085c21f0c4f18f100e3e053b5ceedbed
Signed-off-by: Drew Walters <andrew.walters@att.com>
- With bionic image based shipyard docker images, uwsgi crashes
with segmentation fault, when it tries to load the psycopg2 library,
causing the api become unreachable on both shipyard docker images.
This happens because psycopg2 2.7.x and uwsgi binary wheels are built
with incompatible ssl libraries. This patch upgrades psycopg2 to the
latest release to address this issue.
- The existing image build script cannot run in a docker or a pod,
based pipeline because of two reasons:
- The build script runs a docker (docker-in-docker) and mounts a
volume.
In a dind case, volume bind mounts will not work, because the nested
container will need the host file system's path for the source path.
- The shipyard service listens to its exposed service port in the
nested docker network namespace, which is not reachable from the host
pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.
Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
The airskiff job is looking for a node named "Ubuntu-Bionic", but the
only node available in its nodeset is "primary". This change updates the
playbook to use the proper name.
Change-Id: Ib3a8de0918c7a9e5abb7fb71a20ae261f68b3259
Signed-off-by: Drew Walters <andrew.walters@att.com>
The latest Werkzeug package release 1.0.0, released on February 8,
is not backwards compatible with the earler releases of this package,
which is used in Flask, used by apache airflow.
This causes shipyard make image job fail, with missing import errors
from the Werkzeug library.
This change locks the Werkzeuz package release to the last compatible
release with the apache airflow in shipyard.
Change-Id: I54dad4ccc1858f4d5986c6e8e9fbf8f5d9847158
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
The patch introduces network policy configuration similar
to openstack-helm services. It allows users to configure
policies depending on the environment.
* Network policies are disabled by default.
* When enabled default policies allow all ingress and
egress traffic (i.e. policy set to {}), this may be
changed in future patch-sets.
Change-Id: Ic0b44eb142445d45d81e3e546d394e1c7b451238
Depth 1 clone does not pull the pinned htk version, so resetting
to that version was failing, leaving master in place.
Change-Id: Ice638d429b7051023a381e97df1334d406903f9e
Updated airflow config and Dockerfiles for apache airflow 1.10.4
For details see: https://github.com/apache/airflow/blob/master/UPDATING.md
Also updated the kubectl to 1.16.2, as part of kubernetes 1.16 uplift.
Change-Id: Ib24ff0304b6279ff0be749141854d6a604473597
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.
Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
Update apiversion for ClusterRole, ClusterRoleBinding to rbac.authorization.k8s.io/v1
Update apiversion for deployment to apps/v1
Update apiversion for statefulset to apps/v1
Add selector match labels to deployment
This patch is similar to https://review.opendev.org/#/c/638276/
These changes are required to install shipyard helm chart on k8s 1.16.0
Change-Id: I7ac6fc060fbd6a5feea747ebbe8121c5a2eb4b6f
- Adds the information related to the test_site action.
- Reformats, slightly, the output from 'shipyard help actions'
- Adds tests that use an externalized list of actions to keep the help
documentation in alignment with the actions supported in the API.
Change-Id: I2efd473da0dbf6c8cbadfc9fae575c303996c43b
Updated deckhand commit id, because apache airflow release 1.10.3 and
later use flask >=1.1.0, which needs Werkzeug library version >=0.15.0.
see: https://issues.apache.org/jira/browse/AIRFLOW-4900
The updated deckhand commit lets the Werkzeug package float to >=0.15.0.
Change-Id: I62d7f4e5eecb2f05035a1c9552544aa65e70ee3f
The location of the Airskiff site recently changed [0], causing the
Airskiff job to fail. This change updates the "reduce site" playbook to
match the job in treasuremap, thereby avoiding the directory which no
longer exists.
The change also removes the usage of install-packages script, because
it was removed [1].
[0] https://review.opendev.org/674963
[1] https://review.opendev.org/672540
Change-Id: I7b20ff7c50cfa085039e893558df2cf022c4333c
Signed-off-by: Drew Walters <andrew.walters@att.com>
- Allow a Docker build-arg to specify the source of the Airflow
python package in any format supported by pip
Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
This patch makes Celery to connect to RabbitMQ directly instead
of using LB. It also brings a forked version of a transport url
template, the reason for this is the format for Kombu/Celery
broker url is different from oslo_messaging transport url:
1. URLs need to be separated with semicolons vs commas.
2. Every item in Kombu broker url needs to be a complete url
that includes schema, vhost, and all credentials.
This format is specific to Airflow and is not used in upstream
OSH projects, hence it is included here and not in htk.
Depends-On: I5150a64bd29fa062e30496c1f2127de138322863
Change-Id: I0b4ae6a9538f2f6988ed42c8f5cf0a54e7a7ad2e
Pegleg Promenade and Shipyard should all use same versions of
packages when able. Requests currently is giving a warning
in Pegleg:
ERROR: deckhand 0.0.1.dev657 has requirement urllib3==1.24, but
you'll have urllib3 1.24.3 which is incompatible.
Change-Id: I9f21203b4109e7542a952d68c7a6ffcdb2653026
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: I09b5d45fb5ca505b1f97d0a78cc082c1e0254edd
airflow.cfg file was mounted as a dir, not a file,
so airflow service doesn't want to start.
This reverts commit 6794903558495e288f9cb6794e6459409de1fc71.
Change-Id: I6db528ac91fc5cb6719831eb2915467105f4c491
Recently, the airflow config mounts were changed to projected volumes to
workaround a K8s bug [0]; however, a subpath prevents the configs from
being properly mounted. This change removes the subpath.
[0] https://review.opendev.org/671944
Change-Id: I9bbe91d3e27b293a6fd27c00545329bc8a36f926
Signed-off-by: Drew Walters <andrew.walters@att.com>
Because of a kubernetes bug [0] when a container which
is mounted with the subpath option, the configmap is
changed and then the container restarts the mounting of
the configmap fails.
This PS uses the projected key for volume definitions
as a workaround.
[0] https://github.com/kubernetes/kubernetes/issues/68211
Change-Id: I6820a0f963c5b28e1674ea58214ffc86009db4dd
Setuptools is currently listed as a requirement for shipyard-client, but
it is not used. Pegging setuptools forces downgrading when installing
shipyard-client, which can cause issues for packages that attempt to
utilize features found in newer versions of setuptools. Removing the
setuptools requirements will allow installing users to choose their own
version of setuptools to use.
Change-Id: I88570b69346ec7019e753bdd8681fd63d55dc8c8
Since ':master' and ':latest' Shipyard and Airflow images are outdated,
set defaults to Ubuntu Xenial -based images.
Change-Id: I40978747f31c6a8c5cc8689a9768f8c4836ac1a1
