791 Commits

Author SHA1 Message Date
Sergiy Markin
7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin
9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin
296853f159 Uplift airship modules
This PS uplifts airship dependent modules.

Change-Id: Ib17cd12c6b88fbb618af68a2112e315c2cc9c0fd
2023-05-10 13:19:44 +00:00
Sergiy Markin
b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Sergiy Markin
154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Ruslan Aliev
e3e71f7d16 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I3bc287af7a89410b7e7e248673a854477360f9ac
2022-09-28 09:51:39 -05:00
Zuul
aba1bb1355 Merge "[airflow] Bumping up kubectl version to 1.24.6" 2022-09-26 21:16:03 +00:00
Zuul
f4142cb03f Merge "[shipyard] Postgres upgrade to 14.5" 2022-09-26 21:16:02 +00:00
Zuul
34ac308ff2 Merge "[helm-toolkit] Update to latest helm-toolkit" 2022-09-26 21:12:02 +00:00
Markin, Sergiy (sm515x)
a3d60cfbc8 [shipyard] Fix pyOpenSSL issue in xenial images
Change-Id: I4c57f92b56a2956497bf6466e23afdfb9af8080f
2022-09-23 16:32:06 +00:00
Markin, Sergiy (sm515x)
e79867956b [airflow] Bumping up kubectl version to 1.24.6
Change-Id: I93613f02a4d86e8776c64b6f7756f21bc1cb95ff
2022-09-23 14:25:28 +00:00
Markin, Sergiy (sm515x)
ed5775c0ed [shipyard] Postgres upgrade to 14.5
This PS upgrades shipyard postgres to 14.5

Change-Id: I1524cca569015aa7c4068fbaa6cf9b3221821941
2022-09-23 13:14:12 +00:00
Markin, Sergiy (sm515x)
2fb7b5371b [helm-toolkit] Update to latest helm-toolkit
Change-Id: Ia6cc049d2f1526b8e6bc13f98c2867d6d9e39ee7
2022-09-23 01:13:26 +00:00
Markin, Sergiy (sm515x)
ad19720444 [zuul] Added a flush node firewall task to Airskiff gates
Airskiff gates are setting up a k8s cluster by minikube, so test node needs a clear firewall rules in order to let calico to get initialized. So a task that flushes firewall rules and  deploy-package role have been added to ansible in gates.

Change-Id: I19f9c8a28b394fa239313bda8b008d79cc048469
2022-09-22 23:18:46 +00:00
Markin, Sergiy (sm515x)
d49d4ce52b [openssl] Bump up version of OpenSSL in requirements
OpenSSL has been updated to 22.0.0 in order to image build process.

Change-Id: I8f3a81439c7fea47d5769ed343a20d273052e4bf
2022-09-22 21:56:14 +00:00
Ruslan Aliev
8240608fe5 Loose pyyaml requirements
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I2a3cf7b3af74db3331329931761b5686988f0ac0
2022-07-14 15:12:47 -05:00
Ruslan Aliev
9a8218aa25 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I61d65874864f233372df72787cdb25a54b2edbc3
2022-07-07 14:14:51 -05:00
Phil Sphicas
7dcada80f1 Uplift Armada client, enable socket keepalives
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.

This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)

Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
2022-03-21 13:54:38 -07:00
Chris Wedgwood
034b906dd6 [airflow] fix ordering of affinity/psp in charts
Change-Id: I08afee40ac3965adc2bcff7f1a2a21ebb39fb87c
2021-11-03 14:08:25 -05:00
Phil Sphicas
aa99869546 Update HTK stable commit (Ingress)
Update helm-toolkit to include updated Ingress templates that use
current API versions (i.e. networking.k8s.io/v1)

https://review.opendev.org/c/openstack/openstack-helm-infra/+/813115

Change-Id: I7878ec1ecfe63e84155e69b0902d5c0612c867da
2021-10-18 12:43:13 -07:00
Phil Sphicas
0b985d60a9 Gate fixes: remove py35, make xenial non-voting
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.

Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
2021-10-18 12:43:13 -07:00
Sean Eagan
e64f17b91b Remove Helm 2 based Armada integrations
- Tiller status check
- Test cleanup arg removed

Depends-On: https://review.opendev.org/c/airship/armada/+/812047
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I77ef3fb8e952ad28132e3476138d34bbb5a6fd3d
2021-10-04 15:03:04 -05:00
Sean Eagan
a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Sean Eagan
27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
anthony.bellino
3a27007e8a (zuul) Fix Shipyard Post Gates
Change-Id: I9f6f633399069f28c47c381faf4230e4f22276e2
2021-09-10 13:13:38 -07:00
Maximilian Weiss
88286bc690 Gate fixes for Shipyard
* Fixed typo for jsonschema version

* Setuptools version forced to <58 to allow usage of use_2to3, which
  some dependencies require

* Fixed scripting error that allowed Airflow installation to silently
  fail

Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
2021-09-09 20:23:05 +00:00
Maximilian Weiss
ac236ef52a Update HTK stable commit to 0.2.19
Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/c/openstack/openstack-helm-infra/+/802925

Change-Id: I07e82486c901b558ab2f9619d5663593305c504f
2021-08-17 17:50:35 +00:00
Maximilian Weiss
99e253092a Update helm installation script
Updates the helm installation script to download and install v2.17.0
from get.helm.sh (instead of v2.16.9 from storage.googleapis.com).

Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
2021-08-17 17:50:35 +00:00
anthony.bellino
00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
  in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Rick Bartra
35e5a7f796 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
2020-09-30 01:19:51 +00:00
Zuul
fb1c281b83 Merge "Accelerate YAML operations with LibYAML" 2020-09-28 17:12:04 +00:00
Zuul
24d301851a Merge "Include LibYAML in container builds" 2020-09-28 17:01:55 +00:00
Zuul
19a11a7766 Merge "Fix airflow quicktest error" 2020-09-28 17:01:54 +00:00
Andrii Ostapenko
af63252e8d
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5
2020-09-24 19:43:13 -05:00
Phil Sphicas
4e7ff997a6 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
2020-09-24 05:09:46 +00:00
Phil Sphicas
6b5f9c4db4 Include LibYAML in container builds
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.

https://pyyaml.org/wiki/LibYAML

Change-Id: I6ebafa1eb2af8841b86b64fd119dcb581e6c18a7
2020-09-24 04:35:26 +00:00
Phil Sphicas
86d2e0d9f2 Fix airflow quicktest error
airflow quicktest is throwing an error when trying to start the
webserver:
    can't open file '_cmd': [Errno 2] No such file or directory

Change-Id: I3fddc1cdd3daeacc09a5924287d711a72878c5e3
2020-09-24 04:35:26 +00:00
Rick Bartra
b117fadbdb fix: Docker image build jobs
Update the deb-docker path to fix the docker image build jobs

Change-Id: I31660fcc5358fc67caba39fdaf38b60b13e7eaff
2020-09-22 09:23:18 -05:00
KAVVA, JAGAN MOHAN REDDY (jk330k)
796f2ddcca Move Tiller version to 2.16.9
Update Helm chart for Shipyard to use Tiller version 2.16.9.

Change-Id: Ia61098db57c741f36864db084d39dd9b27d25251
2020-09-16 17:00:54 +00:00
Zuul
8b69cebeea Merge "fix: Gate fix for Shipyard docs and pep8 gates" 2020-09-15 19:52:36 +00:00
Rick Bartra
d38078cce3 fix: Gate fix for Shipyard docs and pep8 gates
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.

Additionally, address pep8 violations that arise from using a newer
version of flake8

Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
2020-09-15 18:16:45 +00:00
Bartra, Rick (rb560u)
1ae9a7c4be Improve logging of Shipyard _execute_task method during failure or time out
In the case where a Drydock task either fails or times out, it is not very
clear which Drydock task is being referred to. This commit updates the log
messages to include the Drydock task-id that has either failed or timed out.
Additionally, debug logging is added to log the Drydock task status. The reason
for this is because there are situations where a Drydock task is still running,
but on the Shipyard side the task has timed out and therefore, Shipyard shows
the task (i.e. prepare_nodes or deploy_nodes) as failed. It is good to get a point
in time reference to the task state for a couple of reasons:
1) To verify via logging that Shipyard is reporting Drydock tasks (success,
   failures, partial_successes, etc) accurately
2) Later on if Drydock shows a node as deployed, then the task can be
   queried and the state can be checked to see if the task was indeed a success.

Change-Id: I7050338e2f92ad548e639e2ea4059a520c27e686
2020-08-28 14:00:13 -04:00
Mahmoudi, Ahmad (am495p)
a5e57879ab Override uwsgi default config
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
  performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
  improvements since 2018.

Upgraded uwsgi to bring in bug fixes since 2018.

For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
2020-08-06 02:14:07 +00:00
KHIYANI, RAHUL (rk0850)
841d0ee2c8 Add the missing readOnly-fs flag for airflow-web containers
Change-Id: Ic31288d2f88c9610621ef1b74d8de813ea985ca8
2020-07-17 16:00:26 +00:00
KHIYANI, RAHUL (rk0850)
db37122336 Implement helm-toolkit snippet to airflow pods/containers
This updates the airflow chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag

Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
2020-07-09 20:36:39 +00:00
Zuul
66d410779c Merge "Add configmap-hash annotations for Shipyard & Airflow" 2020-07-06 21:48:15 +00:00
DODDA, PRATEEK REDDY
5247fed4ba Add configmap-hash annotations for Shipyard & Airflow
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.

These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.

Change-Id: I59eb516086c4fd41f7c18923f86f135101656af8
2020-07-06 12:09:18 -05:00
KHIYANI, RAHUL (rk0850)
02929cfc44 Implement helm-toolkit snippet to shipyard pods/containers
This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag to true

Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
2020-07-02 09:25:11 -05:00
Zuul
f6e4066a83 Merge "Enabling Apparmor profile to shipyard init containers" 2020-06-26 19:27:14 +00:00
Zuul
83501c83c2 Merge "Uplift celery to address memory leak" 2020-06-26 16:46:54 +00:00