starlingx
/
integ
Code Issues Proposed changes

Debian: Upversion armada sources for container build 

Currently, we use in the playbooks [1] an armada image built upstream
[2]. We use armada from upstream helm2 branch.

Armada needs to be patched to add support for k8s >=1.22.
Proposed an upstream patch, but we don't know how long it takes until
it is merged:
https://review.opendev.org/c/airship/armada/+/845392

Instead of waiting for upstream commit to merge, and an image be
generated, we provide the code change here, so an armada image with
k8s >=1.22 support can be generated. The k8s >=1.22 support is added
by patch 0003.
Necessary StarlingX build changes to support generating an container
image and push to https://hub.docker.com/r/starlingx will be
addressed in another commit.

Note: since we always used an upstream built armada container, this
package purpose was to provide helm chart overrides. We add a new
purpose: to release to opensource the changes we are about to do to
armada, since we'll be building a container image using these changes.

To achieve this we do the following:
- upversion sources from 7ef4b8643b5ec5216a8f6726841e156c0aa54a1a
to ddbdd7256c20f138737f6cbd772312f7a19f58b8. This ensures we are
patching the image used in the playbooks[2].
- create patches 0001 and 0002 to ensure there are no helm chart
changes between upversion. This reduces testing effort related to
original purpose (provide helm chart overrides) of this package.
- create patch 0003 to add k8s >=1.22 support.
- old patches are not changed, but renamed from 0001->0005 to
0004->0008 and regenerated.

Other notes:
We don't need to port this work to CentOS. This work is supposed to be
temporary until the upstream airship/armada commit merges.

Tests on Debian:
PASS: build-pkgs -c -p armada
PASS: make images
      Upload image to controller, use it.
      Using the new armada image do an apply,remove,apply,remove,apply
      chain for a custom StarlingX app.

[1]: https://opendev.org/starlingx/ansible-playbooks/src/branch/
master/playbookconfig/src/playbooks/roles/common/
load-images-information/vars/k8s-v1.22.5/system-images.yml#L5
[2] quay.io/airshipit/armada:
ddbdd7256c20f138737f6cbd772312f7a19f58b8-ubuntu_bionic

Partial-Bug: 1978409
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Change-Id: Id51c241a3965ef462d325da4ffce37a81693a9f4
This commit is contained in:
Dan Voiculeasa 2022-06-14 16:10:32 +03:00
parent 8dd7885e4e
commit 0783ec950b
10 changed files with 348 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
kubernetes/armada/debian/deb_folder/patches/0001-Revert-Tiller-listen-on-localhost-by-default.patch Normal file
View File

@ -0,0 +1,83 @@
From bf0cfeb9efe5c021b24dcd5ef4c353507d96e307 Mon Sep 17 00:00:00 2001
From: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date: Tue, 14 Jun 2022 15:45:22 +0300
Subject: [PATCH 1/8] Revert "Tiller: listen on localhost by default"
This reverts commit a3f11e5873bc5b97de579c627d7b57e3bc9f655e.
Updating the sources from 7ef4b8643b5ec5216a8f6726841e156c0aa54a1a to
ddbdd7256c20f138737f6cbd772312f7a19f58b8 we observe a change to default
tiller port logic.
Keep old logic.
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
---
charts/armada/templates/deployment-api.yaml | 2 +-
charts/armada/values.yaml | 6 ------
charts/tiller/templates/deployment-tiller.yaml | 2 +-
charts/tiller/values.yaml | 6 ------
4 files changed, 2 insertions(+), 14 deletions(-)
diff --git a/charts/armada/templates/deployment-api.yaml b/charts/armada/templates/deployment-api.yaml
index b7f93d7..f1395ef 100644
--- a/charts/armada/templates/deployment-api.yaml
+++ b/charts/armada/templates/deployment-api.yaml
@@ -178,7 +178,7 @@ spec:
{{- end }}
{{- end }}
- -listen
- - "{{ if not .Values.conf.tiller.listen_on_any }}127.0.0.1{{ end }}:{{ .Values.conf.tiller.port }}"
+ - ":{{ .Values.conf.tiller.port }}"
- -probe-listen
- ":{{ .Values.conf.tiller.probe_port }}"
- -logtostderr
diff --git a/charts/armada/values.yaml b/charts/armada/values.yaml
index b1ed61f..3a4427e 100644
--- a/charts/armada/values.yaml
+++ b/charts/armada/values.yaml
@@ -206,12 +206,6 @@ conf:
tiller:
# If set to false then some form of Tiller needs to be provided
enabled: true
- # To have Tiller bind to all interfaces, allowing direct connections from
- # the Helm client to pod_ip:port, set 'listen_on_any: true'.
- # The default setting 'listen_on_any: false' binds Tiller to 127.0.0.1.
- # The Armada container talks directly to Tiller via 127.0.0.1, so the
- # default value is appropriate for normal operation.
- listen_on_any: false
port: 24134
probe_port: 24135
verbosity: 5
diff --git a/charts/tiller/templates/deployment-tiller.yaml b/charts/tiller/templates/deployment-tiller.yaml
index f6df614..1f04a13 100644
--- a/charts/tiller/templates/deployment-tiller.yaml
+++ b/charts/tiller/templates/deployment-tiller.yaml
@@ -104,7 +104,7 @@ spec:
{{- end }}
{{- end }}
- -listen
- - "{{ if not .Values.conf.tiller.listen_on_any }}127.0.0.1{{ end }}:{{ .Values.conf.tiller.port }}"
+ - ":{{ .Values.conf.tiller.port }}"
- -probe-listen
- ":{{ .Values.conf.tiller.probe_port }}"
- -logtostderr
diff --git a/charts/tiller/values.yaml b/charts/tiller/values.yaml
index ba776bc..495e3c1 100644
--- a/charts/tiller/values.yaml
+++ b/charts/tiller/values.yaml
@@ -52,12 +52,6 @@ conf:
# Note: Defaulting to the (default) kubernetes grace period, as anything
# greater than that will have no effect.
prestop_sleep: 30
- # To have Tiller bind to all interfaces, allowing direct connections from
- # the Helm client to pod_ip:port, set 'listen_on_any: true'.
- # The default setting 'listen_on_any: false' binds Tiller to 127.0.0.1.
- # Helm clients with Kubernetes API access dynamically set up a portforward
- # into the pod, which works with the default setting.
- listen_on_any: false
port: 44134
probe_port: 44135
--
2.34.1

32
kubernetes/armada/debian/deb_folder/patches/0002-Revert-Add-labels-to-Armada-deployment.patch Normal file
View File

@ -0,0 +1,32 @@
From 6d63302d8e7d35c4549c49b97c8667203bc22428 Mon Sep 17 00:00:00 2001
From: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date: Tue, 14 Jun 2022 16:01:56 +0300
Subject: [PATCH 2/8] Revert "Add "labels" to Armada deployment"
This reverts commit eb2e87d32b2b9c9853deb70ed2e7029380ef0e16.
Updating the sources from 7ef4b8643b5ec5216a8f6726841e156c0aa54a1a to
ddbdd7256c20f138737f6cbd772312f7a19f58b8.
This revert is here just to reduce the need for testing. Keeping the
helm charts without changes between 7ef4b8..ddbdd7.
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
---
charts/armada/templates/deployment-api.yaml | 2 --
1 file changed, 2 deletions(-)
diff --git a/charts/armada/templates/deployment-api.yaml b/charts/armada/templates/deployment-api.yaml
index f1395ef..562e3d0 100644
--- a/charts/armada/templates/deployment-api.yaml
+++ b/charts/armada/templates/deployment-api.yaml
@@ -92,8 +92,6 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: armada-api
- labels:
-{{ $labels | indent 4 }}
annotations:
{{ tuple $envAll | include "helm-toolkit.snippets.release_uuid" | indent 4 }}
spec:
--
2.34.1

205
kubernetes/armada/debian/deb_folder/patches/0003-Create-lock-CRD-as-apiextensions.k8s.io-v1-object.patch Normal file
View File

@ -0,0 +1,205 @@
From 9c37292171aa9c35fbfb8c1ee2670150b9621190 Mon Sep 17 00:00:00 2001
From: Phil Sphicas <phil.sphicas@att.com>
Date: Wed, 9 Feb 2022 10:04:38 -0800
Subject: [PATCH 3/8] Create lock CRD as apiextensions.k8s.io/v1 object
Cherry-pick https://review.opendev.org/c/airship/armada/+/845392
from upstream keeping the original commit message:
-----
Kubernetes v1.22 stopped serving the apiextensions.k8s.io/v1beta1 API
version of CustomResourceDefinition.
This change ensures that the locks.armada.process CRD is created using
the apiextensions.k8s.io/v1 API.
The kubernetes client package is also updated to take advantage of the
dynamic client.
(cherry picked from commit c5d39f27cacaa953be43c7d3265bb693db0939d0)
In addition to the clean cherry-pick add fixes for tests:
- docs language
- protobuf version
- stestr missing
Closes-Bug: 1978409
Change-Id: Icd518ab5cbb78e8b15f63d19c51b5f5b9a67e995
-----
On top of the upstream cherry-pick we need to enchance build env to
add missing .yaml files.
Change setup.py and add MANIFEST.in to allow proper contents of the
image to be generated.
Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
---
MANIFEST.in | 2 ++
armada/handlers/k8s.py | 7 +++----
armada/handlers/lock.py | 42 +++++++++++++++++------------------------
doc/source/conf.py | 2 +-
requirements.txt | 6 +++---
setup.py | 2 +-
test-requirements.txt | 1 +
7 files changed, 28 insertions(+), 34 deletions(-)
create mode 100644 MANIFEST.in
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..61709ba
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,2 @@
+include *.yaml
+recursive-include armada *.yaml
diff --git a/armada/handlers/k8s.py b/armada/handlers/k8s.py
index 31b00c7..56cbc26 100644
--- a/armada/handlers/k8s.py
+++ b/armada/handlers/k8s.py
@@ -57,8 +57,7 @@ class K8s(object):
self.batch_api = client.BatchV1Api(api_client)
self.batch_v1beta1_api = client.BatchV1beta1Api(api_client)
self.custom_objects = client.CustomObjectsApi(api_client)
- self.api_extensions = client.ApiextensionsV1beta1Api(api_client)
- self.extension_api = client.ExtensionsV1beta1Api(api_client)
+ self.api_extensions = client.ApiextensionsV1Api(api_client)
self.apps_v1_api = client.AppsV1Api(api_client)
def delete_job_action(
@@ -359,10 +358,10 @@ class K8s(object):
:param crd: custom resource definition to create
- :type crd: kubernetes.client.V1beta1CustomResourceDefinition
+ :type crd: kubernetes.client.V1CustomResourceDefinition
:return: new custom resource definition
- :rtype: kubernetes.client.V1beta1CustomResourceDefinition
+ :rtype: kubernetes.client.V1CustomResourceDefinition
"""
return self.api_extensions.create_custom_resource_definition(crd)
diff --git a/armada/handlers/lock.py b/armada/handlers/lock.py
index bd99d38..54e3eb0 100644
--- a/armada/handlers/lock.py
+++ b/armada/handlers/lock.py
@@ -281,40 +281,32 @@ class LockConfig:
return lock
def create_definition(self):
- names = client.V1beta1CustomResourceDefinitionNames(
+ names = client.V1CustomResourceDefinitionNames(
kind="Resource", plural=LOCK_PLURAL, singular=LOCK_SINGULAR)
metadata = client.V1ObjectMeta(
name="{}.{}".format(LOCK_PLURAL, LOCK_GROUP),
resource_version=LOCK_VERSION)
- status = client.V1beta1CustomResourceDefinitionStatus(
- accepted_names=names,
- conditions=[],
- stored_versions=[LOCK_VERSION])
- spec = client.V1beta1CustomResourceDefinitionSpec(
+ spec = client.V1CustomResourceDefinitionSpec(
group=LOCK_GROUP,
names=names,
scope="Namespaced",
- version=LOCK_VERSION)
- crd = client.V1beta1CustomResourceDefinition(
- spec=spec,
- status=status,
- metadata=metadata,
- kind="CustomResourceDefinition")
+ versions=[
+ {
+ "name": LOCK_VERSION,
+ "schema": {
+ "openAPIV3Schema": {
+ "type": "object",
+ "x-kubernetes-preserve-unknown-fields": True
+ }
+ },
+ "served": True,
+ "storage": True,
+ }
+ ])
+ crd = client.V1CustomResourceDefinition(
+ spec=spec, metadata=metadata, kind="CustomResourceDefinition")
try:
self.k8s.create_custom_resource_definition(crd)
- except ValueError as err:
- # Because of an issue with the Kubernetes code, the API server
- # may return `null` for the required field `conditions` in
- # kubernetes.client.V1beta1CustomResourceDefinitionStatus
- # This causes validation to fail which will raise the subsequent
- # ValueError even though the CRD was created successfully
- # https://github.com/kubernetes-client/gen/issues/52
- # TODO if this is fixed upstream this should be removed
- known_msg = "Invalid value for `conditions`, must not be `None`"
- known_err = ValueError(known_msg)
- if err.args != known_err.args:
- raise
- LOG.debug("Encountered known issue while creating CRD, continuing")
except ApiException as err:
# If a 409 is received then the definition already exists
if err.status != 409:
diff --git a/doc/source/conf.py b/doc/source/conf.py
index 6ed6273..b4826c7 100644
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -78,7 +78,7 @@ author = 'The Airship Authors'
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
diff --git a/requirements.txt b/requirements.txt
index 2d61ca3..c2f9ac2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -6,16 +6,16 @@ jsonschema>=3.0.1,<4
keystoneauth1>=3.18.0
keystonemiddleware==5.3.0
kombu<4.7,>=4.6.10
-kubernetes>=12.0.0
+kubernetes<23,>=17.0.0
Paste>=2.0.3
PasteDeploy>=1.5.2
-protobuf>=3.4.0
+protobuf>=3.4.0,<3.21
pylibyaml~=0.1
pyyaml~=5.1
requests
retry
setuptools>=40.4.3
-prometheus_client>=0.7.0
+prometheus_client<0.13.0,>=0.7.0
# API
falcon
diff --git a/setup.py b/setup.py
index 7d9b694..8a5a815 100644
--- a/setup.py
+++ b/setup.py
@@ -5,4 +5,4 @@ try:
except ImportError:
pass
-setuptools.setup(setup_requires=['pbr>=2.0.0'], pbr=True)
+setuptools.setup(setup_requires=['pbr>=2.0.0'], pbr=True, include_package_data=True)
diff --git a/test-requirements.txt b/test-requirements.txt
index 3108c65..e9faa28 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -16,6 +16,7 @@ mock
responses>=0.8.1
yapf==0.27.0
flake8-import-order==0.18.1
+stestr>=1.0.0 # Apache-2.0
grpcio-tools==1.16.0
typing-extensions==3.7.2
--
2.34.1

6
kubernetes/armada/debian/deb_folder/patches/0001-Add-Helm-v2-client-initialization.patch → kubernetes/armada/debian/deb_folder/patches/0004-Add-Helm-v2-client-initialization.patch
View File

@ -1,7 +1,7 @@
From 0dad89825be15c596d5678d8daafa671f859f4f6 Mon Sep 17 00:00:00 2001
From 1c73f6739eb672b330669fda5e427099c08c3490 Mon Sep 17 00:00:00 2001
From: Thiago Brito <thiago.brito@windriver.com>
Date: Thu, 22 Apr 2021 20:00:51 -0300
Subject: [PATCH] Add Helm v2 client initialization
Subject: [PATCH 4/8] Add Helm v2 client initialization
This adds helm v2 client initialization using the tiller
container postStart exec to access helm v2 binary.
@ -108,5 +108,5 @@ index 3a4427e..da45810 100644
# Should be the `$HOME/.kube` of the `runAsUser` above
# as this is where tiller's kubernetes client roots its cache dir.
--
2.25.1
2.34.1

8
kubernetes/armada/debian/deb_folder/patches/0002-Tiller-wait-for-postgres-database-ping.patch → kubernetes/armada/debian/deb_folder/patches/0005-Tiller-wait-for-postgres-database-ping.patch
View File

@ -1,7 +1,7 @@
From 96e49fcc6d6b988d03a61261511abf64a0af2e2a Mon Sep 17 00:00:00 2001
From 0cd23c208587d86ce8b2083bf4f42dadf03e28e2 Mon Sep 17 00:00:00 2001
From: Dan Voiculeasa <dan.voiculeasa@windriver.com>
Date: Tue, 11 May 2021 21:04:18 +0300
Subject: [PATCH] Tiller wait for postgres database ping
Subject: [PATCH 5/8] Tiller wait for postgres database ping
Networking might not be correctly initialized when tiller starts.
@ -14,7 +14,7 @@ Signed-off-by: Dan Voiculeasa <dan.voiculeasa@windriver.com>
1 file changed, 19 insertions(+), 12 deletions(-)
diff --git a/charts/armada/templates/deployment-api.yaml b/charts/armada/templates/deployment-api.yaml
index 69036c0..3816366 100644
index 69036c0..bf23fb2 100644
--- a/charts/armada/templates/deployment-api.yaml
+++ b/charts/armada/templates/deployment-api.yaml
@@ -167,24 +167,31 @@ spec:
@ -62,5 +62,5 @@ index 69036c0..3816366 100644
postStart:
exec:
--
2.30.0
2.34.1

6
kubernetes/armada/debian/deb_folder/patches/0003-Update-the-liveness-probe-to-verify-postgres-connect.patch → kubernetes/armada/debian/deb_folder/patches/0006-Update-the-liveness-probe-to-verify-postgres-connect.patch
View File

@ -1,7 +1,7 @@
From be3167e5342f2730ef43012d8fe4f3782c6ef468 Mon Sep 17 00:00:00 2001
From 3e34ec8c87ec5cbac79f6299e63fa1c06a75692e Mon Sep 17 00:00:00 2001
From: Robert Church <robert.church@windriver.com>
Date: Wed, 12 May 2021 02:38:52 -0400
Subject: [PATCH 3/3] Update the liveness probe to verify postgres connectivity
Subject: [PATCH 6/8] Update the liveness probe to verify postgres connectivity
Change the tillerLivenessProbeTemplate to test the connectivity to the
postgres backend. We will override the periodSeconds and
@ -41,5 +41,5 @@ index bf23fb2..2b65494 100644
{{- if .Values.manifests.deployment_api }}
--
2.16.6
2.34.1

6
kubernetes/armada/debian/deb_folder/patches/0004-Update-postgres-liveness-check-to-support-IPv6-addre.patch → kubernetes/armada/debian/deb_folder/patches/0007-Update-postgres-liveness-check-to-support-IPv6-addre.patch
View File

@ -1,7 +1,7 @@
From e13416638b103fde04feb31027c3148c9685cf7f Mon Sep 17 00:00:00 2001
From c5a117faafbe1cc6de4200315ba400bdc0c83e40 Mon Sep 17 00:00:00 2001
From: Robert Church <robert.church@windriver.com>
Date: Sat, 15 May 2021 16:16:41 -0400
Subject: [PATCH 4/4] Update postgres liveness check to support IPv6 addresses
Subject: [PATCH 7/8] Update postgres liveness check to support IPv6 addresses
Templating will add square brackets for IPv6 addresses which are
interpreted as an array vs. a string. Quote this so that it interpreted
@ -26,5 +26,5 @@ index 2b65494..5c4825c 100644
{{- end }}
--
2.16.6
2.34.1

6
kubernetes/armada/debian/deb_folder/patches/0005-Add-toleration-to-armada-api.patch → kubernetes/armada/debian/deb_folder/patches/0008-Add-toleration-to-armada-api.patch
View File

@ -1,7 +1,7 @@
From a2e44bc741366c39ebae7add1564a8ac67f13895 Mon Sep 17 00:00:00 2001
From 9a5b91d35923272c55e998f21ff096b5c0268ad2 Mon Sep 17 00:00:00 2001
From: Enzo Candotti <enzo.candotti@windriver.com>
Date: Wed, 6 Oct 2021 18:25:10 -0300
Subject: [PATCH] Add toleration to armada-api
Subject: [PATCH 8/8] Add toleration to armada-api
Signed-off-by: Yue Tao <Yue.Tao@windriver.com>
---
@ -54,5 +54,5 @@ index da45810..3644706 100644
type: apparmor
armada-api:
--
2.25.1
2.34.1

13
kubernetes/armada/debian/deb_folder/patches/series
View File

@ -1,5 +1,8 @@
0001-Add-Helm-v2-client-initialization.patch
0002-Tiller-wait-for-postgres-database-ping.patch
0003-Update-the-liveness-probe-to-verify-postgres-connect.patch
0004-Update-postgres-liveness-check-to-support-IPv6-addre.patch
0005-Add-toleration-to-armada-api.patch
0001-Revert-Tiller-listen-on-localhost-by-default.patch
0002-Revert-Add-labels-to-Armada-deployment.patch
0003-Create-lock-CRD-as-apiextensions.k8s.io-v1-object.patch
0004-Add-Helm-v2-client-initialization.patch
0005-Tiller-wait-for-postgres-database-ping.patch
0006-Update-the-liveness-probe-to-verify-postgres-connect.patch
0007-Update-postgres-liveness-check-to-support-IPv6-addre.patch
0008-Add-toleration-to-armada-api.patch

8
kubernetes/armada/debian/meta_data.yaml
View File

@ -2,10 +2,10 @@
debname: armada
debver: 0.2.0-0
dl_path:
name: armada-7ef4b8643b5ec5216a8f6726841e156c0aa54a1a.tar.gz
url: https://github.com/airshipit/armada/tarball/7ef4b8643b5ec5216a8f6726841e156c0aa54a1a
md5sum: 58e2ee93d7e177e96e2cf0de1873ccce
sha256sum: fb3763d28430f5602136a23de0abf1880273c804fe7f13696d8078cea615f7b4
name: armada-ddbdd7256c20f138737f6cbd772312f7a19f58b8.tar.gz
url: https://github.com/airshipit/armada/tarball/ddbdd7256c20f138737f6cbd772312f7a19f58b8
md5sum: fd8563e106a48b912b56ac82e8a5d4ee
sha256sum: 4f4db518837f7e89e3d65e024353f49ea341c530533cb986ff50d0781f39bee6
revision:
dist: $STX_DIST
PKG_GITREVCOUNT: true