Support hyperkube_prefix label

Additionally for k8s_fedora_coreos_v1 driver:
* Introduce hyperkube_prefix which defaults to k8s.gcr.io/
* Bump default kube_tag to v1.18.16

Story: 1668998
Task: 41791

Change-Id: I38b8df45a00f1a2a1604059b8329d1dd762e05cd
This commit is contained in:
Bharat Kunwar 2020-09-16 14:19:31 +00:00
parent 61cb34528d
commit fc1f27a569
17 changed files with 77 additions and 16 deletions

View File

@ -342,6 +342,8 @@ the table are linked to more details elsewhere in the user guide.
+---------------------------------------+--------------------+---------------+
| `grafana_admin_passwd`_ | (any string) | "admin" |
+---------------------------------------+--------------------+---------------+
| `hyperkube_prefix`_ | see below | see below |
+---------------------------------------+--------------------+---------------+
| `kube_tag`_ | see below | see below |
+---------------------------------------+--------------------+---------------+
| `cloud_provider_tag`_ | see below | see below |
@ -1268,6 +1270,15 @@ _`container_infra_prefix`
* quay.io/k8scsi/csi-resizer:v0.3.0
* quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
_`hyperkube_prefix`
This label allows users to specify a custom prefix for Hyperkube container
source since official Hyperkube images have been discontinued for `kube_tag`
greater than 1.18.x. If you wish you use 1.19.x onwards, you may want to use
unofficial sources like `docker.io/rancher/`, `ghcr.io/openstackmagnum/` or
your own container registry. If `container_infra_prefix` label is defined, it
still takes precedence over this label.
Default: k8s.gcr.io/
_`kube_tag`
This label allows users to select a specific Kubernetes release based on its
container tag for `Fedora Atomic
@ -1280,6 +1291,7 @@ _`kube_tag`
Stein default: v1.11.6
Train default: v1.15.7
Ussuri default: v1.18.2
Victoria default: v1.18.16
_`heapster_enabled`
heapster_enabled is used to enable disable the installation of heapster.

View File

@ -93,7 +93,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-apiserver \\
--volume /etc/ssl/certs:/etc/ssl/certs:ro \\
--volume /run:/run \\
--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kube-apiserver \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBE_ETCD_SERVERS \$KUBE_API_ADDRESS \$KUBELET_PORT \$KUBE_SERVICE_ADDRESSES \$KUBE_ADMISSION_CONTROL \$KUBE_API_ARGS'
ExecStop=-/usr/bin/podman stop kube-apiserver
@ -122,7 +122,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-controller-manager \\
--volume /etc/ssl/certs:/etc/ssl/certs:ro \\
--volume /run:/run \\
--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kube-controller-manager \\
--secure-port=0 \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBE_MASTER \$KUBE_CONTROLLER_MANAGER_ARGS'
@ -152,7 +152,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-scheduler \\
--volume /etc/ssl/certs:/etc/ssl/certs:ro \\
--volume /run:/run \\
--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kube-scheduler \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBE_MASTER \$KUBE_SCHEDULER_ARGS'
ExecStop=-/usr/bin/podman stop kube-scheduler
@ -208,7 +208,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
--volume /var/run/lock:/var/run/lock:z \\
--volume /opt/cni/bin:/opt/cni/bin:z \\
--volume /etc/machine-id:/etc/machine-id \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kubelet \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBELET_API_SERVER \$KUBELET_ADDRESS \$KUBELET_PORT \$KUBELET_HOSTNAME \$KUBELET_ARGS'
ExecStop=-/usr/bin/podman stop kubelet
@ -241,7 +241,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
--volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
--volume /lib/modules:/lib/modules:ro \\
--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kube-proxy \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBE_MASTER \$KUBE_PROXY_ARGS'
ExecStop=-/usr/bin/podman stop kube-proxy

View File

@ -106,7 +106,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
--volume /var/run/lock:/var/run/lock:z \\
--volume /opt/cni/bin:/opt/cni/bin:z \\
--volume /etc/machine-id:/etc/machine-id \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kubelet \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBELET_API_SERVER \$KUBELET_ADDRESS \$KUBELET_PORT \$KUBELET_HOSTNAME \$KUBELET_ARGS'
ExecStop=-/usr/bin/podman stop kubelet
@ -139,7 +139,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
--volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
--volume /lib/modules:/lib/modules:ro \\
--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
\${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:\${KUBE_TAG} \\
\${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
kube-proxy \\
\$KUBE_LOGTOSTDERR \$KUBE_LOG_LEVEL \$KUBE_MASTER \$KUBE_PROXY_ARGS'
ExecStop=-/usr/bin/podman stop kube-proxy

View File

@ -36,7 +36,7 @@ do
done
if [ "$(echo $USE_PODMAN | tr '[:upper:]' '[:lower:]')" == "true" ]; then
KUBE_DIGEST=$($ssh_cmd podman image inspect ${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:${KUBE_TAG} --format "{{.Digest}}")
KUBE_DIGEST=$($ssh_cmd podman image inspect ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG} --format "{{.Digest}}")
if [ -n "${KUBE_IMAGE_DIGEST}" ] && [ "${KUBE_IMAGE_DIGEST}" != "${KUBE_DIGEST}" ]; then
printf "The sha256 ${KUBE_DIGEST} of current hyperkube image cannot match the given one: ${KUBE_IMAGE_DIGEST}."
exit 1

View File

@ -6,7 +6,7 @@ set +x
. /etc/sysconfig/heat-params
set -x
hyperkube_image="${CONTAINER_INFRA_PREFIX:-"k8s.gcr.io/"}hyperkube:${KUBE_TAG}"
hyperkube_image="${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG}"
ssh_cmd="ssh -F /srv/magnum/.ssh/config root@localhost"
mkdir -p /srv/magnum/bin/
i=0

View File

@ -37,7 +37,7 @@ if [ "${new_kube_tag}" != "${KUBE_TAG}" ]; then
${ssh_cmd} podman rm ${service}
done
${ssh_cmd} podman rmi ${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:${KUBE_TAG}
${ssh_cmd} podman rmi ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG}
echo "KUBE_TAG=$new_kube_tag" >> /etc/sysconfig/heat-params
for service in ${SERVICE_LIST}; do
@ -45,7 +45,7 @@ if [ "${new_kube_tag}" != "${KUBE_TAG}" ]; then
done
i=0
until [ "`${ssh_cmd} podman image exists ${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:${new_kube_tag} && echo $?`" = 0 ]
until [ "`${ssh_cmd} podman image exists ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${new_kube_tag} && echo $?`" = 0 ]
do
i=$((i+1))
[ $i -lt 30 ] || break;
@ -53,7 +53,7 @@ if [ "${new_kube_tag}" != "${KUBE_TAG}" ]; then
sleep 5s
done
KUBE_DIGEST=$($ssh_cmd podman image inspect ${CONTAINER_INFRA_PREFIX:-k8s.gcr.io/}hyperkube:${new_kube_tag} --format "{{.Digest}}")
KUBE_DIGEST=$($ssh_cmd podman image inspect ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${new_kube_tag} --format "{{.Digest}}")
if [ -n "${new_kube_image_digest}" ] && [ "${new_kube_image_digest}" != "${KUBE_DIGEST}" ]; then
printf "The sha256 ${KUBE_DIGEST} of current hyperkube image cannot match the given one: ${new_kube_image_digest}."
exit 1

View File

@ -68,6 +68,7 @@ REGION_NAME="$REGION_NAME"
HTTP_PROXY="$HTTP_PROXY"
HTTPS_PROXY="$HTTPS_PROXY"
NO_PROXY="$NO_PROXY"
HYPERKUBE_PREFIX="$HYPERKUBE_PREFIX"
KUBE_TAG="$KUBE_TAG"
CLOUD_PROVIDER_TAG="$CLOUD_PROVIDER_TAG"
CLOUD_PROVIDER_ENABLED="$CLOUD_PROVIDER_ENABLED"

View File

@ -56,6 +56,7 @@ HTTP_PROXY="$HTTP_PROXY"
HTTPS_PROXY="$HTTPS_PROXY"
NO_PROXY="$NO_PROXY"
WAIT_CURL="$WAIT_CURL"
HYPERKUBE_PREFIX="$HYPERKUBE_PREFIX"
KUBE_TAG="$KUBE_TAG"
FLANNEL_NETWORK_CIDR="$FLANNEL_NETWORK_CIDR"
PODS_NETWORK_CIDR="$PODS_NETWORK_CIDR"

View File

@ -75,6 +75,7 @@ class K8sFedoraTemplateDefinition(k8s_template_def.K8sTemplateDefinition):
extra_params['cloud_provider_enabled'] = cloud_provider_enabled
label_list = ['coredns_tag',
'hyperkube_prefix',
'kube_tag', 'container_infra_prefix',
'availability_zone',
'cgroup_driver',

View File

@ -444,6 +444,11 @@ parameters:
type: string
description: url for keystone
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
default: k8s.gcr.io/
kube_tag:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
@ -1183,6 +1188,7 @@ resources:
http_proxy: {get_param: http_proxy}
https_proxy: {get_param: https_proxy}
no_proxy: {get_param: no_proxy}
hyperkube_prefix: {get_param: hyperkube_prefix}
kube_tag: {get_param: master_kube_tag}
cloud_provider_tag: {get_param: cloud_provider_tag}
cloud_provider_enabled: {get_param: cloud_provider_enabled}
@ -1411,6 +1417,7 @@ resources:
http_proxy: {get_param: http_proxy}
https_proxy: {get_param: https_proxy}
no_proxy: {get_param: no_proxy}
hyperkube_prefix: {get_param: hyperkube_prefix}
kube_tag: {get_param: minion_kube_tag}
kube_version: {get_param: kube_version}
trustee_user_id: {get_param: trustee_user_id}

View File

@ -255,6 +255,10 @@ parameters:
type: string
description: no proxies for docker
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
kube_tag:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
@ -755,6 +759,7 @@ resources:
"$HTTP_PROXY": {get_param: http_proxy}
"$HTTPS_PROXY": {get_param: https_proxy}
"$NO_PROXY": {get_param: no_proxy}
"$HYPERKUBE_PREFIX": {get_param: hyperkube_prefix}
"$KUBE_TAG": {get_param: kube_tag}
"$CLOUD_PROVIDER_TAG": {get_param: cloud_provider_tag}
"$CLOUD_PROVIDER_ENABLED": {get_param: cloud_provider_enabled}

View File

@ -196,6 +196,10 @@ parameters:
type: string
description: no proxies for docker
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
kube_tag:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
@ -434,6 +438,7 @@ resources:
$HTTP_PROXY: {get_param: http_proxy}
$HTTPS_PROXY: {get_param: https_proxy}
$NO_PROXY: {get_param: no_proxy}
$HYPERKUBE_PREFIX: {get_param: hyperkube_prefix}
$KUBE_TAG: {get_param: kube_tag}
$FLANNEL_NETWORK_CIDR: {get_param: flannel_network_cidr}
$PODS_NETWORK_CIDR: {get_param: pods_network_cidr}

View File

@ -446,26 +446,31 @@ parameters:
type: string
description: url for keystone
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
default: k8s.gcr.io/
kube_tag:
type: string
description:
tag of the k8s containers used to provision the kubernetes cluster
https://github.com/kubernetes/kubernetes/releases
default: v1.18.2
default: v1.18.16
master_kube_tag:
type: string
description:
tag of the k8s containers used to provision the kubernetes cluster
https://github.com/kubernetes/kubernetes/releases
default: v1.18.2
default: v1.18.16
minion_kube_tag:
type: string
description:
tag of the k8s containers used to provision the kubernetes cluster
https://github.com/kubernetes/kubernetes/releases
default: v1.18.2
default: v1.18.16
cloud_provider_tag:
type: string
@ -501,7 +506,7 @@ parameters:
kube_version:
type: string
description: version of kubernetes used for kubernetes cluster
default: v1.18.2
default: v1.18.16
kube_dashboard_version:
type: string
@ -1211,6 +1216,7 @@ resources:
http_proxy: {get_param: http_proxy}
https_proxy: {get_param: https_proxy}
no_proxy: {get_param: no_proxy}
hyperkube_prefix: {get_param: hyperkube_prefix}
kube_tag: {get_param: master_kube_tag}
cloud_provider_tag: {get_param: cloud_provider_tag}
cloud_provider_enabled: {get_param: cloud_provider_enabled}
@ -1442,6 +1448,7 @@ resources:
http_proxy: {get_param: http_proxy}
https_proxy: {get_param: https_proxy}
no_proxy: {get_param: no_proxy}
hyperkube_prefix: {get_param: hyperkube_prefix}
kube_tag: {get_param: minion_kube_tag}
kube_version: {get_param: kube_version}
trustee_user_id: {get_param: trustee_user_id}

View File

@ -259,6 +259,10 @@ parameters:
type: string
description: no proxies for docker
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
kube_tag:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
@ -774,6 +778,7 @@ resources:
"$HTTP_PROXY": {get_param: http_proxy}
"$HTTPS_PROXY": {get_param: https_proxy}
"$NO_PROXY": {get_param: no_proxy}
"$HYPERKUBE_PREFIX": {get_param: hyperkube_prefix}
"$KUBE_TAG": {get_param: kube_tag}
"$CLOUD_PROVIDER_TAG": {get_param: cloud_provider_tag}
"$CLOUD_PROVIDER_ENABLED": {get_param: cloud_provider_enabled}

View File

@ -200,6 +200,10 @@ parameters:
type: string
description: no proxies for docker
hyperkube_prefix:
type: string
description: prefix to use for hyperkube images
kube_tag:
type: string
description: tag of the k8s containers used to provision the kubernetes cluster
@ -459,6 +463,7 @@ resources:
$HTTP_PROXY: {get_param: http_proxy}
$HTTPS_PROXY: {get_param: https_proxy}
$NO_PROXY: {get_param: no_proxy}
$HYPERKUBE_PREFIX: {get_param: hyperkube_prefix}
$KUBE_TAG: {get_param: kube_tag}
$FLANNEL_NETWORK_CIDR: {get_param: flannel_network_cidr}
$PODS_NETWORK_CIDR: {get_param: pods_network_cidr}

View File

@ -477,6 +477,7 @@ class AtomicK8sTemplateDefinitionTestCase(BaseK8sTemplateDefinitionTestCase):
'boot_volume_size')
etcd_volume_size = mock_cluster.labels.get(
'etcd_volume_size')
hyperkube_prefix = mock_cluster.labels.get('hyperkube_prefix')
kube_tag = mock_cluster.labels.get('kube_tag')
etcd_tag = mock_cluster.labels.get('etcd_tag')
coredns_tag = mock_cluster.labels.get('coredns_tag')
@ -641,6 +642,7 @@ class AtomicK8sTemplateDefinitionTestCase(BaseK8sTemplateDefinitionTestCase):
'username': 'fake_user',
'magnum_url': mock_osc.magnum_url.return_value,
'region_name': mock_osc.cinder_region_name.return_value,
'hyperkube_prefix': hyperkube_prefix,
'kube_tag': kube_tag,
'etcd_tag': etcd_tag,
'coredns_tag': coredns_tag,
@ -1008,6 +1010,7 @@ class AtomicK8sTemplateDefinitionTestCase(BaseK8sTemplateDefinitionTestCase):
'boot_volume_size')
etcd_volume_size = mock_cluster.labels.get(
'etcd_volume_size')
hyperkube_prefix = mock_cluster.labels.get('hyperkube_prefix')
kube_tag = mock_cluster.labels.get('kube_tag')
etcd_tag = mock_cluster.labels.get('etcd_tag')
coredns_tag = mock_cluster.labels.get('coredns_tag')
@ -1177,6 +1180,7 @@ class AtomicK8sTemplateDefinitionTestCase(BaseK8sTemplateDefinitionTestCase):
'region_name': mock_osc.cinder_region_name.return_value,
'loadbalancing_protocol': 'HTTP',
'kubernetes_port': 8080,
'hyperkube_prefix': hyperkube_prefix,
'kube_tag': kube_tag,
'etcd_tag': etcd_tag,
'coredns_tag': coredns_tag,

View File

@ -0,0 +1,8 @@
---
features:
- |
Support `hyperkube_prefix` label which defaults to k8s.gcr.io/. Users now
have the option to define alternative hyperkube image source since the
default source has discontinued publication of hyperkube images for
`kube_tag` greater than 1.18.x. Note that if `container_infra_prefix` label
is define, it still takes precedence over this label.