zuul/zuul
Code Issues Proposed changes

Merge "Implement k8s namespaces"

Browse Source
This commit is contained in:
Zuul
2026-06-30 15:54:12 +00:00
committed by Gerrit Code Review
parent 340626e326 941c7058ad
commit b527deed28
7 changed files with 425 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/drivers/kubernetes-attrs.rstinc
+214 -33
View File
@@ -514,7 +514,7 @@
metadata. The value of this field is added to the
`metadata.annotations` field in Kubernetes. This field
contains arbitrary key/value pairs that can be accessed by
tools and libraries.
tools and libraries.
.. attr:: boot-timeout
:type: int
@@ -644,6 +644,34 @@
The time (in seconds) to wait for a snapshot to complete.
.. attr:: spec
:type: dict
Zuul will supply the contents of this value verbatim to
Kubernetes as the ``spec`` attribute of the Kubernetes
``Pod`` definition.
This attribute allows for the creation of arbitrary
complex pod definitions but the user is responsible for
ensuring that they are suitable. The first container in
the pod is expected to be a long-running container that
hosts a shell environment for running commands. The
following minimal definition is recommended as a starting
point:
.. code-block:: yaml
labels:
- name: custom-pod
kind: pod
spec:
containers:
- name: custom-pod
image: ubuntu:jammy
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["sleep infinity"]
.. attr:: tags
:type: dict
@@ -652,10 +680,15 @@
values.
.. attr:: labels
:type: dict
:type: list
A list of labels associated with this provider.
.. attr:: labels[pod]
:type: dict
These are the attributes available for a pod label.
.. attr:: aliases
:type: str
@@ -672,7 +705,7 @@
metadata. The value of this field is added to the
`metadata.annotations` field in Kubernetes. This field
contains arbitrary key/value pairs that can be accessed by
tools and libraries.
tools and libraries.
.. attr:: boot-timeout
:type: int
@@ -754,7 +787,7 @@
.. attr:: kind
The Kubernetes driver supports two types of labels:
The type of label:
.. value:: pod
@@ -762,6 +795,183 @@
pod and a service account that can exec and get the
logs of the pod.
.. attr:: max-age
:type: int
:default: 0
The time (in seconds) since creation that a node may be
available for use. Ready nodes older than this time will be
deleted.
.. attr:: max-ready-age
:type: int
:default: 0
The time (in seconds) an unassigned node should stay in ready state.
.. attr:: min-retention-time
:type: int
:default: 0
The time (in seconds) since an instance was launched, during
which a node will not be deleted. For node resources with
minimum billing times, this can be used to ensure that the
instance is retained for at least the minimum billing interval.
This setting takes precedence over `max-[ready-]age`.
.. attr:: name
:type: str
The name of the label. Used to refer to the label in Zuul
configuration.
.. attr:: reuse
:type: bool
:default: False
Should the node be reused (True) or deleted (False) after use.
.. attr:: slots
:type: int
:default: 1
How many jobs are permitted run on the same node simultaneously.
.. attr:: snapshot-expiration
:type: int
:default: 604800
The time (in seconds) until a snapshot expires.
.. attr:: snapshot-timeout
:type: int
:default: 3600
The time (in seconds) to wait for a snapshot to complete.
.. attr:: spec
:type: dict
Zuul will supply the contents of this value verbatim to
Kubernetes as the ``spec`` attribute of the Kubernetes
``Pod`` definition.
This attribute allows for the creation of arbitrary
complex pod definitions but the user is responsible for
ensuring that they are suitable. The first container in
the pod is expected to be a long-running container that
hosts a shell environment for running commands. The
following minimal definition is recommended as a starting
point:
.. code-block:: yaml
labels:
- name: custom-pod
kind: pod
spec:
containers:
- name: custom-pod
image: ubuntu:jammy
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["sleep infinity"]
.. attr:: tags
:type: dict
A dictionary of tags to add to nodes. Avoid the use of
`zuul_` as a key prefix since Zuul uses this for internal
values.
.. attr:: labels[namespace]
:type: dict
These are the attributes available for a namespace label.
.. attr:: boot-timeout
:type: int
:default: 300
The time (in seconds) to wait for a node to boot.
.. attr:: description
:type: str
A textual description of the label for reference purposes.
.. attr:: executor-zone
:type: str
Specify that a Zuul executor in the specified zone is
used to run jobs with nodes from this label.
.. attr:: final
:default: False
Whether the configuration of the label may be updated
by values in label-defaults or overidden with a new definition
by sections or providers lower in the hierarchy than the point
at which the final attribute is applied.
.. value:: True
The label may not be updated or overidden.
.. value:: False
The label may be updated or overidden.
.. value:: allow-override
The label may not be updated by label-defaults
but may be explicitly overidden by redefining
it in a new 'label' entry.
.. attr:: flavor
:type: str
The flavor to use with this label.
.. attr:: image
:type: str
The image to use with this label.
.. attr:: image-pull-secrets
:type: dict
The imagePullSecrets needed to pull container images from
a private registry. Because Zuul creates pods in a new
namespace, and image pull secrets must exist in the
namespace of the pods that use them, the referenced
secrets will be copied into the temporary namespace that
Zuul creates before creating the pod. The new secrets
will have the same name as the old secrets.
Each entry is a dictionary with the following keys:
.. attr:: name
:type: str
Identifier for this secret. The referenced secret must
already exist under this name so that Nodepool may copy
it. It will be copied into the new namespace with the
same name, therefore, if multiple entries are provided,
they must have distinct names.
.. attr:: namespace
:type: str
:default: default
The namespace of the existing secret to copy.
.. attr:: kind
The type of label:
.. value:: namespace
Namespace labels provide an empty namespace configured
@@ -823,35 +1033,6 @@
The time (in seconds) to wait for a snapshot to complete.
.. attr:: spec
:type: dict
Zuul will supply the contents of this value verbatim to
Kubernetes as the ``spec`` attribute of the Kubernetes
``Pod`` definition.
This attribute allows for the creation of arbitrary
complex pod definitions but the user is responsible for
ensuring that they are suitable. The first container in
the pod is expected to be a long-running container that
hosts a shell environment for running commands. The
following minimal definition is recommended as a starting
point:
.. code-block:: yaml
labels:
- name: custom-pod
kind: pod
spec:
containers:
- name: custom-pod
image: ubuntu:jammy
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["sleep infinity"]
.. attr:: tags
:type: dict
tests/fixtures/layouts/kubernetes/nodepool.yaml
Vendored
+7
View File
@@ -96,6 +96,11 @@
image: debian
flavor: normal
- label:
name: k8s-namespace
image: debian
flavor: normal
- section:
name: kube-base
abstract: true
@@ -127,3 +132,5 @@
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["while true; do sleep 30; done;"]
- name: k8s-namespace
kind: namespace
tests/unit/test_kubernetes_driver.py
+24 -1
View File
@@ -34,6 +34,7 @@ from tests.base import (
BaseTestCase,
FIXTURE_DIR,
ZuulTestCase,
driver_config,
iterate_timeout,
okay_tracebacks,
simple_layout,
@@ -128,6 +129,18 @@ class BaseKubernetesDriverTest(ZuulTestCase):
'KubernetesProviderEndpoint._completeApi', return_value=None)):
yield
def check_pod_node_attrs(self, pnode):
self.assertIsNotNone(pnode.kubernetes_connection['pod'])
self.assertIsNotNone(pnode.kubernetes_connection['namespace'])
self.assertEqual(pnode.kubernetes_connection['pod'],
pnode.kubernetes_connection['namespace'])
self.assertEqual('kubectl', pnode.connection_type)
def check_namespace_node_attrs(self, pnode):
self.assertIsNone(pnode.kubernetes_connection['pod'])
self.assertIsNotNone(pnode.kubernetes_connection['namespace'])
self.assertEqual('kubectl', pnode.connection_type)
class TestKubernetesDriver(BaseKubernetesDriverTest, BaseCloudDriverTest):
def _assertProviderNodeAttributes(self, pnode):
@@ -138,9 +151,19 @@ class TestKubernetesDriver(BaseKubernetesDriverTest, BaseCloudDriverTest):
checks(self, pnode)
@simple_layout('layouts/kubernetes/nodepool.yaml', enable_nodepool=True)
def test_kubernetes_node_lifecycle(self):
@driver_config('kubernetes',
node_checks=BaseKubernetesDriverTest.
check_pod_node_attrs)
def test_kubernetes_node_lifecycle_pod(self):
self._test_node_lifecycle('debian-normal')
@simple_layout('layouts/kubernetes/nodepool.yaml', enable_nodepool=True)
@driver_config('kubernetes',
node_checks=BaseKubernetesDriverTest.
check_namespace_node_attrs)
def test_kubernetes_node_lifecycle_namespace(self):
self._test_node_lifecycle('k8s-namespace')
@simple_layout('layouts/kubernetes/more.yaml', enable_nodepool=True)
def test_kubernetes_node_lifecycle_more(self):
# Test with more options than the normal test
tools/print_schema.py
+5
View File
@@ -168,6 +168,11 @@ class SchemaWalker:
disc = alt['value']['type']['alts'][0]['value']
altname = f"{name}[{disc}]"
out += self._attr(indent, altname, alt)
# Or k8s label types
elif name == 'labels':
disc = alt['value']['kind']['alts'][0]['value']
altname = f"{name}[{disc}]"
out += self._attr(indent, altname, alt)
else:
out += self._toDoc(indent + 1, alt)
return out
zuul/driver/kubernetes/kubernetesendpoint.py
+11 -4
View File
@@ -121,7 +121,10 @@ class KubernetesCreateStateMachine(statemachine.StateMachine):
not self.endpoint.core_client.api_client.configuration.verify_ssl
self.node.kubernetes_connection['namespace'] = self.namespace
# Pod and namespace names are the same
self.node.kubernetes_connection['pod'] = self.namespace
if label.kind == 'pod':
self.node.kubernetes_connection['pod'] = self.namespace
elif label.kind == 'namespace':
self.node.kubernetes_connection['pod'] = None
self.node.kubernetes_connection['containers'] = self.containers
def advance(self):
@@ -149,9 +152,13 @@ class KubernetesCreateStateMachine(statemachine.StateMachine):
self.state = self.RBAC_CREATING
if self.state == self.RBAC_CREATING:
self.endpoint._createPod(self.namespace, self.hostname,
self.label, self.node.tags)
self.state = self.POD_CREATING
if self.label.kind == 'pod':
self.endpoint._createPod(self.namespace, self.hostname,
self.label, self.node.tags)
self.state = self.POD_CREATING
elif self.label.kind == 'namespace':
self._updateNode()
self.state = self.COMPLETE
if self.state == self.POD_CREATING:
if not self.endpoint._getPod(self.namespace, self.hostname):
zuul/driver/kubernetes/kubernetesprovider.py
+158 -60
View File
@@ -23,6 +23,7 @@ from zuul.lib.voluputil import (
Optional,
Required,
assemble,
discriminate,
)
import voluptuous as vs
@@ -74,11 +75,30 @@ class KubernetesProviderLabel(BaseProviderLabel):
): str,
})
kubernetes_label_schema = vs.Schema({
kubernetes_common_label_schema = vs.Schema({
Optional(
'image-pull-secrets',
default=[],
doc="""\
The imagePullSecrets needed to pull container images from
a private registry. Because Zuul creates pods in a new
namespace, and image pull secrets must exist in the
namespace of the pods that use them, the referenced
secrets will be copied into the temporary namespace that
Zuul creates before creating the pod. The new secrets
will have the same name as the old secrets.
Each entry is a dictionary with the following keys:
""",
): AsList(kubernetes_pull_secrets),
})
kubernetes_label_non_inheritable_schema = vs.Schema({
Required(
'kind',
doc="""\
The Kubernetes driver supports two types of labels:
The type of label:
""",
): vs.Any(
Constant(
@@ -98,43 +118,13 @@ class KubernetesProviderLabel(BaseProviderLabel):
""",
),
),
Required(
'spec',
doc="""\
Zuul will supply the contents of this value verbatim to
Kubernetes as the ``spec`` attribute of the Kubernetes
``Pod`` definition.
This attribute allows for the creation of arbitrary
complex pod definitions but the user is responsible for
ensuring that they are suitable. The first container in
the pod is expected to be a long-running container that
hosts a shell environment for running commands. The
following minimal definition is recommended as a starting
point:
.. code-block:: yaml
labels:
- name: custom-pod
kind: pod
spec:
containers:
- name: custom-pod
image: ubuntu:jammy
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["sleep infinity"]
""",
): dict,
})
kubernetes_label_inheritable_schema = vs.Schema({
Optional(
'kind',
doc="""\
The Kubernetes driver supports two types of labels:
The type of label:
""",
): Nullable(vs.Any(
Constant(
@@ -156,7 +146,74 @@ class KubernetesProviderLabel(BaseProviderLabel):
)),
})
kubernetes_label_common_schema = vs.Schema({
kubernetes_pod_kind_schema = vs.Schema({
Required(
'kind',
doc="""\
The type of label:
""",
): vs.Any(
Constant(
'pod',
doc="""\
Pod labels provide a dedicated namespace with a single
pod and a service account that can exec and get the
logs of the pod.
""",
),
),
})
kubernetes_label_kind_schema = vs.Schema({
Required(
'kind',
doc="""\
The type of label:
""",
): vs.Any(
Constant(
'namespace',
doc="""\
Namespace labels provide an empty namespace configured
with a service account that can create pods, services,
configmaps, etc.
""",
),
),
})
kubernetes_spec_doc = """\
Zuul will supply the contents of this value verbatim to
Kubernetes as the ``spec`` attribute of the Kubernetes
``Pod`` definition.
This attribute allows for the creation of arbitrary
complex pod definitions but the user is responsible for
ensuring that they are suitable. The first container in
the pod is expected to be a long-running container that
hosts a shell environment for running commands. The
following minimal definition is recommended as a starting
point:
.. code-block:: yaml
labels:
- name: custom-pod
kind: pod
spec:
containers:
- name: custom-pod
image: ubuntu:jammy
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["sleep infinity"]
"""
kubernetes_inheritable_pod_label_schema_extra = vs.Schema({
Optional(
'spec',
doc=kubernetes_spec_doc,
): Nullable(dict),
Optional(
'annotations',
doc="""\
@@ -164,42 +221,83 @@ class KubernetesProviderLabel(BaseProviderLabel):
metadata. The value of this field is added to the
`metadata.annotations` field in Kubernetes. This field
contains arbitrary key/value pairs that can be accessed by
tools and libraries. """,
): Nullable(dict),
Optional(
'image-pull-secrets',
default=[],
doc="""\
The imagePullSecrets needed to pull container images from
a private registry. Because Zuul creates pods in a new
namespace, and image pull secrets must exist in the
namespace of the pods that use them, the referenced
secrets will be copied into the temporary namespace that
Zuul creates before creating the pod. The new secrets
will have the same name as the old secrets.
Each entry is a dictionary with the following keys:
tools and libraries.
""",
): AsList(kubernetes_pull_secrets),
): Nullable(dict),
})
kubernetes_non_inheritable_pod_label_schema_extra = vs.Schema({
Required(
'spec',
doc=kubernetes_spec_doc,
): dict,
Optional(
'annotations',
doc="""\
A dictionary of additional values to be added to the pod
metadata. The value of this field is added to the
`metadata.annotations` field in Kubernetes. This field
contains arbitrary key/value pairs that can be accessed by
tools and libraries.
""",
): Nullable(dict),
})
kubernetes_pod_label_schema = assemble(
BaseProviderLabel.schema,
kubernetes_label_non_inheritable_schema,
kubernetes_common_label_schema,
kubernetes_non_inheritable_pod_label_schema_extra,
kubernetes_pod_kind_schema,
doc="These are the attributes available for a pod label.",
)
internal_kubernetes_pod_label_schema = assemble(
BaseProviderLabel.schema,
provider_schema.internal_base_label,
kubernetes_label_non_inheritable_schema,
kubernetes_common_label_schema,
kubernetes_non_inheritable_pod_label_schema_extra,
kubernetes_pod_kind_schema,
extra=vs.ALLOW_EXTRA,
)
kubernetes_namespace_label_schema = assemble(
BaseProviderLabel.schema,
kubernetes_label_non_inheritable_schema,
kubernetes_common_label_schema,
kubernetes_label_kind_schema,
doc="These are the attributes available for a namespace label.",
)
internal_kubernetes_namespace_label_schema = assemble(
BaseProviderLabel.schema,
provider_schema.internal_base_label,
kubernetes_label_non_inheritable_schema,
kubernetes_common_label_schema,
kubernetes_label_kind_schema,
extra=vs.ALLOW_EXTRA,
)
inheritable_schema = assemble(
BaseProviderLabel.inheritable_schema,
kubernetes_label_inheritable_schema,
kubernetes_label_common_schema,
kubernetes_common_label_schema,
kubernetes_inheritable_pod_label_schema_extra,
)
schema = assemble(
BaseProviderLabel.schema,
kubernetes_label_schema,
kubernetes_label_common_schema,
schema = vs.Union(
kubernetes_pod_label_schema,
kubernetes_namespace_label_schema,
discriminant=discriminate(
lambda val, alt: val['kind'] == alt['kind'].validators[0])
)
internal_schema = assemble(
schema,
provider_schema.internal_base_label,
extra=vs.ALLOW_EXTRA,
internal_schema = vs.Union(
internal_kubernetes_pod_label_schema,
internal_kubernetes_namespace_label_schema,
discriminant=discriminate(
lambda val, alt: val['kind'] == alt['kind'].validators[0])
)
image_flavor_inheritable_schema = vs.Schema({})
zuul/executor/server.py
+6 -1
View File
@@ -3037,7 +3037,12 @@ class AnsibleJob(object):
'namespace': data['namespace'],
'context': data['context_name'],
}
if node.connection_type in ('project', 'namespace'):
if (
# Nodepool:
(node.connection_type in ('project', 'namespace')) or
# NIZ:
data['pod'] is None
):
# Project are special nodes that are not the inventory
pass
else: