* Fixed typo for jsonschema version
* Setuptools version forced to <58 to allow usage of use_2to3, which
some dependencies require
* Fixed scripting error that allowed Airflow installation to silently
fail
Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
Updates the helm installation script to download and install v2.17.0
from get.helm.sh (instead of v2.16.9 from storage.googleapis.com).
Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0
Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.
https://pyyaml.org/wiki/LibYAML
Change-Id: I6ebafa1eb2af8841b86b64fd119dcb581e6c18a7
airflow quicktest is throwing an error when trying to start the
webserver:
can't open file '_cmd': [Errno 2] No such file or directory
Change-Id: I3fddc1cdd3daeacc09a5924287d711a72878c5e3
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.
Additionally, address pep8 violations that arise from using a newer
version of flake8
Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
In the case where a Drydock task either fails or times out, it is not very
clear which Drydock task is being referred to. This commit updates the log
messages to include the Drydock task-id that has either failed or timed out.
Additionally, debug logging is added to log the Drydock task status. The reason
for this is because there are situations where a Drydock task is still running,
but on the Shipyard side the task has timed out and therefore, Shipyard shows
the task (i.e. prepare_nodes or deploy_nodes) as failed. It is good to get a point
in time reference to the task state for a couple of reasons:
1) To verify via logging that Shipyard is reporting Drydock tasks (success,
failures, partial_successes, etc) accurately
2) Later on if Drydock shows a node as deployed, then the task can be
queried and the state can be checked to see if the task was indeed a success.
Change-Id: I7050338e2f92ad548e639e2ea4059a520c27e686
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
improvements since 2018.
Upgraded uwsgi to bring in bug fixes since 2018.
For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
This updates the airflow chart to include the pod
security context on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag
Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.
These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.
Change-Id: I59eb516086c4fd41f7c18923f86f135101656af8
This updates the shipyard chart to include the pod
security context on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag to true
Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
Remove OSH Authors copyright
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
Uplifted celery to 4.4.2 to address memory leak issue raised
in: https://github.com/celery/celery/issues/4843.
This issue is addressed in celery 4.4.2
Change-Id: I26c403513ba48308044d69130d33561b314fd209
Updated to pip to pip3 to address zuul gate issue.
Set the ensure_global_symlinks to true for zuul-jobs to set
symlink for tox path."
Change-Id: I1b0634ef18328bccc4d6929072f53db779d70ef1
The Airship vulnerability documentation has moved [0]. This change
updates SECURITY.md to point to the correct location.
[0] https://docs.airshipit.org/learn/vulnerabilities.html
Change-Id: Iea843a3399bc7836f5645c3ca81603e2e9ca7356
Signed-off-by: Drew Walters <andrew.walters@att.com>
The current Armada DAG allows for 3 retries, reduced from 10 here:
bef8eecac1
This is sometimes insufficient, especially in cases where chart updates
of underlying Airship or Kubernetes components are upgraded and pods are
restarted underneath the airflow workers. The chart installation may be
successful, but an Armada retry may still be consumed.
This change increases the number of retries to 5. This will allow Armada
to progress further through the manifest if there is a disruption after
a chart is successfully installed. The tradeoff is that Armada may try
to repeatedly install a chart that keeps failing in the same way,
delaying the ultimate failure of the deployment.
Change-Id: I1fad7b1d95af061595680a76d24c6d323b365a67
1. Locked the WTForms to 2.2.1 to address the import issue with
wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
April 21/2020. This release causes shipyard gate fail with
import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
apache-airflow extra package postgres, and resoted the newer
release psycopg2-binary==2.8.4, to be used instead.
Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
This adds the container security context to set
readOnlyRootFilesystem to true
Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
All Airship projects are moving to GitHub issues. This change adds a
GitHub security policy that links to the official Airship vulnerability
management process [0]. When users on GitHub click "New Issue" on this
GitHub repository, they will see an option to report a security
vulnerability, which will direct them to our official policy.
[0] https://airship-docs.readthedocs.io/en/latest/security/vulnerabilities.html
Change-Id: Iaf060dd0085c21f0c4f18f100e3e053b5ceedbed
Signed-off-by: Drew Walters <andrew.walters@att.com>
- With bionic image based shipyard docker images, uwsgi crashes
with segmentation fault, when it tries to load the psycopg2 library,
causing the api become unreachable on both shipyard docker images.
This happens because psycopg2 2.7.x and uwsgi binary wheels are built
with incompatible ssl libraries. This patch upgrades psycopg2 to the
latest release to address this issue.
- The existing image build script cannot run in a docker or a pod,
based pipeline because of two reasons:
- The build script runs a docker (docker-in-docker) and mounts a
volume.
In a dind case, volume bind mounts will not work, because the nested
container will need the host file system's path for the source path.
- The shipyard service listens to its exposed service port in the
nested docker network namespace, which is not reachable from the host
pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.
Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
The airskiff job is looking for a node named "Ubuntu-Bionic", but the
only node available in its nodeset is "primary". This change updates the
playbook to use the proper name.
Change-Id: Ib3a8de0918c7a9e5abb7fb71a20ae261f68b3259
Signed-off-by: Drew Walters <andrew.walters@att.com>
The latest Werkzeug package release 1.0.0, released on February 8,
is not backwards compatible with the earler releases of this package,
which is used in Flask, used by apache airflow.
This causes shipyard make image job fail, with missing import errors
from the Werkzeug library.
This change locks the Werkzeuz package release to the last compatible
release with the apache airflow in shipyard.
Change-Id: I54dad4ccc1858f4d5986c6e8e9fbf8f5d9847158
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
Sean Eagan