starlingx/stx-puppet
Code Issues Proposed changes

Rework advertise address in apiserver-change-param

Browse Source 
The current implementation assumes that /etc/kubernetes/kubeadm.yaml is
always present on the system, which isn't true when the active host
isn't controller-0.

The new implementation will store the original advertise-address value
and replace it on kube-apiserver config as soon as it gets reset during
kubeadm init phase control-plane.

The downside of this approach is that kube-apiserver is initially
started with the default advertise address value (default network
interface) to then get updated with the previous correct value,
restarting kube-apiserver once more.

Test Plan: Verify advertise-address is not affected during apiserver
change params

Note: advertise-address is found in
`/etc/kubernetes/manifests/kube-apiserver.yaml`

PASS: Verify that modifying and applying kube-apiserver oidc parameters
doesn't affect advertise-address
PASS: Verify upgrade command `system upgrade-activate` doesn't affect
advertise-address
PASS: Verify `/etc/kubernetes/manifests/kube-apiserver.yaml` changes after
apiserver-change-params are consistent
PASS: Verify ConfigMap kube-system kubeadm-config changes are consistent
after apiserver-change-params are consistent after
apiserver-change-params
PASS: IPv6 - Verify that modifying kube-apiserver params doesn't affect
advertise-address

Regression:

PASS: Verify system install
PASS: Verify upgrade completes successfully
PASS: Verify backup and restore completes successfully

Notes: When possible, the above tests were executed in both AIO-SX and
Standard systems, both controllers.

Closes-Bug: 1900153
Signed-off-by: Rafael Camargos <RafaelLucas.Camargos@windriver.com>
Change-Id: I95c5cc277fc343c383f9e4b3942e13f2009e1ab6
This commit is contained in:
Rafael Camargos 2022-01-11 14:16:22 -03:00
parent 8c938fd5be
commit 04a1c1b080
2 changed files with 8 additions and 41 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
puppet-manifests/src/modules/platform/files/append_init_configuration_to_kube_apiserver_cluster_config.py
View File

@ -1,36 +0,0 @@
#
# Copyright (c) 2021 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# This script edits a file containing a kubernetes ClusterConfiguration,
# appending to it the current InitConfiguration retrieved from kubeadm.
# This is especially useful during the command 'kubeadm init phase control-plane
# apiserver' which will reset the advertise-address parameter when called
# without a InitConfiguration.
import argparse
import ruamel.yaml as yaml
INIT_CONFIGURATION = 'InitConfiguration'
parser = argparse.ArgumentParser()
parser.add_argument('--cluster_config_file', required=True)
args = parser.parse_args()
cluster_config_path = args.cluster_config_file
with open(cluster_config_path, 'r') as cluster_config_file:
cluster_config = yaml.load(cluster_config_file,
Loader=yaml.RoundTripLoader)
with open('/etc/kubernetes/kubeadm.yaml', 'r') as kubeadm_file:
kubeadm_config = yaml.load_all(kubeadm_file, Loader=yaml.RoundTripLoader)
init_config = next(config for config in kubeadm_config
if config['kind'] == INIT_CONFIGURATION)
with open(cluster_config_path, 'w') as cluster_config_file:
yaml.dump_all([init_config, cluster_config],
cluster_config_file,
Dumper=yaml.RoundTripDumper,
default_flow_style=False)

13
puppet-manifests/src/modules/platform/templates/kube-apiserver-change-params.erb
View File

@ -1,9 +1,11 @@
<%# Kubeadm stores the cluster configuration as a configmap in the cluster. -%>
<%# We will change that configmap to include/remove kube-apiserver parameters. -%>
<%# In order to restart kube-apiserver, we will use the "kubeadm init phase" -%>
<%# command and feed it the current ClusterConfiguration appended to -%>
<%# InitConfiguration, which contains the kube-apiserver advertise address. -%>
<%# This keeps the configmap consistent and keeps kube-apiserver managed by kubeadm. -%>
<%# command and feed it the current ClusterConfiguration. This will keep the -%>
<%# configmap consistent and keeps kube-apiserver managed by kubeadm, but as a -%>
<%# side-effect, the kube-apiserver advertise address gets reset to the default -%>
<%# network interface. As a work around for that, we'll manually set it back -%>
<%# to its previous value. -%>
umask 077; touch <%= @configmap_temp_file %>
umask 077; touch <%= @configview_temp_file %>
kubectl --kubeconfig=/etc/kubernetes/admin.conf get configmap kubeadm-config -o yaml -n kube-system > <%= @configmap_temp_file %>
@ -37,10 +39,11 @@ python /usr/share/puppet/modules/platform/files/change_kube_apiserver_params.py
--etcd_servers <%= @etcd_servers %>
<%- end -%>
APISERVER_ADVERTISE_ADDRESS=$(grep 'advertise-address=' /etc/kubernetes/manifests/kube-apiserver.yaml | cut -d "=" -f2)
kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system patch configmap kubeadm-config -p "$(cat <%= @configmap_temp_file %>)"
kubectl --kubeconfig=/etc/kubernetes/admin.conf get cm -n kube-system kubeadm-config -o=jsonpath='{.data.ClusterConfiguration}' > <%= @configmap_temp_file %>
python /usr/share/puppet/modules/platform/files/append_init_configuration_to_kube_apiserver_cluster_config.py \
--cluster_config_file <%= @configmap_temp_file %>
kubeadm init phase control-plane apiserver --config <%= @configmap_temp_file %>
DEFAULT_NETWORK_INTERFACE=$(grep 'advertise-address=' /etc/kubernetes/manifests/kube-apiserver.yaml | cut -d "=" -f2)
sed -i "s/$DEFAULT_NETWORK_INTERFACE/$APISERVER_ADVERTISE_ADDRESS/g" /etc/kubernetes/manifests/kube-apiserver.yaml
rm <%= @configmap_temp_file %>
rm <%= @configview_temp_file %>