Fix usage of the trustee user in K8S Cinder plugin

Closes-Bug: #1672667
Change-Id: I702818777ea4664ecd560c4b7a02431c86988e17
changes/04/445404/16
Mathieu Velten 6 years ago
parent 0c76374327
commit c0787b4e94
  1. 55
      doc/source/dev/kubernetes-load-balancer.rst
  2. 51
      doc/source/userguide.rst
  3. 24
      magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
  4. 44
      magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
  5. 1
      magnum/drivers/common/templates/kubernetes/fragments/write-heat-params-master.yaml
  6. 2
      magnum/drivers/common/templates/kubernetes/fragments/write-heat-params.yaml
  7. 9
      magnum/drivers/common/templates/kubernetes/fragments/write-kube-os-config.sh
  8. 13
      magnum/drivers/heat/template_def.py
  9. 1
      magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml
  10. 5
      magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml
  11. 9
      magnum/drivers/k8s_fedora_atomic_v1/templates/kubeminion.yaml
  12. 2
      magnum/tests/unit/conductor/handlers/test_swarm_cluster_conductor.py

@ -78,57 +78,7 @@ neutron_lbaas.conf::
To configure LBaaS v1 or v2, refer to the Neutron documentation.
To enable the load balancer, log into each master node of your cluster and
perform the following steps:
1. Configure kube-apiserver::
sudo vi /etc/kubernetes/apiserver
Comment out the line::
#KUBE_API_ARGS="--runtime_config=api/all=true"
Uncomment the line::
KUBE_API_ARGS="--runtime_config=api/all=true --cloud_config=/etc/sysconfig/kube_openstack_config --cloud_provider=openstack"""
2. Configure kube-controller-manager::
sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
Immediately after the lines::
- controller-manager
- --master=http://127.0.0.1:8080
- --service-account-private-key-file=/etc/kubernetes/ssl/server.key
- --root-ca-file=/etc/kubernetes/ssl/ca.crt
Add the following lines::
- --cloud_config=/etc/sysconfig/kube_openstack_config
- --cloud_provider=openstack
When the file is saved, the pod will automatically restart the
kube-controller-manager container to pick up the change.
3. Enter OpenStack user credential::
sudo vi /etc/sysconfig/kube_openstack_config
The username and tenant-name entries have been filled in with the
Keystone values of the user who created the cluster. Enter the password
of this user on the entry for password::
password=ChangeMe
4. Restart the Kubernetes API server::
sudo service kube-apiserver restart
service kube-apiserver status
This only needs to be done once. The steps can be reversed to disable the
load balancer feature. Before deleting the Kubernetes cluster, make sure to
Before deleting the Kubernetes cluster, make sure to
delete all the services that created load balancers. Because the Neutron
objects created by Kubernetes are not managed by Heat, they will not be
deleted by Heat and this will cause the cluster-delete operation to fail. If
@ -138,6 +88,9 @@ lb-healthmonitor) and then run cluster-delete again.
Steps for the users
===================
This feature requires the OpenStack cloud provider to be enabled.
To do so, enable the cinder support (--volume-driver cinder).
For the user, publishing the service endpoint externally involves the following
2 steps:

@ -2212,12 +2212,8 @@ Following are some examples for using Cinder as persistent storage.
Using Cinder in Kubernetes
++++++++++++++++++++++++++
**NOTE:** This feature requires Kubernetes version 1.1.1 or above and
Docker version 1.8.3 or above. The public Fedora image from Atomic
currently meets this requirement.
**NOTE:** The following steps are a temporary workaround, and Magnum's
development team is working on a long term solution to automate these steps.
**NOTE:** This feature requires Kubernetes version 1.5.0 or above.
The public Fedora image from Atomic currently meets this requirement.
1. Create the ClusterTemplate.
@ -2240,49 +2236,6 @@ development team is working on a long term solution to automate these steps.
--cluster-template k8s-cluster-template \
--node-count 1
3. Configure kubelet.
To allow Kubernetes to interface with Cinder, log into each minion
node of your cluster and perform step 4 through 6::
sudo vi /etc/kubernetes/kubelet
Comment out the line::
#KUBELET_ARGS=--config=/etc/kubernetes/manifests --cadvisor-port=4194
Uncomment the line::
#KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config"
4. Enter OpenStack user credential::
sudo vi /etc/kubernetes/kube_openstack_config
The username, tenant-name and region entries have been filled in with the
Keystone values of the user who created the cluster. Enter the password
of this user on the entry for password::
password=ChangeMe
5. Restart Kubernetes services::
sudo systemctl restart kubelet
On restart, the new configuration enables the Kubernetes cloud provider
plugin for OpenStack, along with the necessary credential for kubelet
to authenticate with Keystone and to make request to OpenStack services.
6. Install nsenter::
sudo docker run -v /usr/local/bin:/target jpetazzo/nsenter
The nsenter utility is used by Kubernetes to run new processes within
existing kernel namespaces. This allows the kubelet agent to manage storage
for pods.
Kubernetes is now ready to use Cinder for persistent storage.
Following is an example illustrating how Cinder is used in a pod.

@ -4,11 +4,6 @@
echo "configuring kubernetes (master)"
if [ -z "$KUBE_NODE_IP" ]; then
# FIXME(yuanying): Set KUBE_NODE_IP correctly
KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
fi
sed -i '
/^KUBE_ALLOW_PRIV=/ s/=.*/="--allow-privileged='"$KUBE_ALLOW_PRIV"'"/
' /etc/kubernetes/config
@ -31,6 +26,10 @@ if [ -n "${ADMISSION_CONTROL_LIST}" ] && [ "${TLS_DISABLED}" == "False" ]; then
KUBE_ADMISSION_CONTROL="--admission-control=${ADMISSION_CONTROL_LIST}"
fi
if [ -n "$TRUST_ID" ]; then
KUBE_API_ARGS="$KUBE_API_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
fi
sed -i '
/^KUBE_API_ADDRESS=/ s/=.*/="'"${KUBE_API_ADDRESS}"'"/
/^KUBE_SERVICE_ADDRESSES=/ s|=.*|="--service-cluster-ip-range='"$PORTAL_NETWORK_CIDR"'"|
@ -39,10 +38,7 @@ sed -i '
/^KUBE_ADMISSION_CONTROL=/ s/=.*/="'"${KUBE_ADMISSION_CONTROL}"'"/
' /etc/kubernetes/apiserver
cat << _EOC_ >> /etc/kubernetes/apiserver
#Uncomment the following line to disable Load Balancer feature
KUBE_API_ARGS="$KUBE_API_ARGS"
#Uncomment the following line to enable Load Balancer feature
#KUBE_API_ARGS="$KUBE_API_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
_EOC_
# Add controller manager args
@ -50,16 +46,18 @@ KUBE_CONTROLLER_MANAGER_ARGS=""
if [ -n "${ADMISSION_CONTROL_LIST}" ] && [ "${TLS_DISABLED}" == "False" ]; then
KUBE_CONTROLLER_MANAGER_ARGS="--service-account-private-key-file=/srv/kubernetes/server.key --root-ca-file=/srv/kubernetes/ca.crt"
fi
if [ -n "$TRUST_ID" ]; then
KUBE_CONTROLLER_MANAGER_ARGS="$KUBE_CONTROLLER_MANAGER_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
fi
sed -i '
/^KUBELET_ADDRESSES=/ s/=.*/="--machines='""'"/
/^KUBE_CONTROLLER_MANAGER_ARGS=/ s#\(KUBE_CONTROLLER_MANAGER_ARGS\).*#\1="'"${KUBE_CONTROLLER_MANAGER_ARGS}"'"#
' /etc/kubernetes/controller-manager
cat << _EOC_ >> /etc/kubernetes/controller-manager
#Uncomment the following line to enable Kubernetes Load Balancer feature
#KUBE_CONTROLLER_MANAGER_ARGS="\$KUBE_CONTROLLER_MANAGER_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
_EOC_
KUBELET_ARGS="--register-node=true --register-schedulable=false --config=/etc/kubernetes/manifests --hostname-override=$KUBE_NODE_IP"
HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
KUBELET_ARGS="--register-node=true --register-schedulable=false --config=/etc/kubernetes/manifests --hostname-override=${HOSTNAME_OVERRIDE}"
if [ -n "${INSECURE_REGISTRY_URL}" ]; then
KUBELET_ARGS="${KUBELET_ARGS} --pod-infra-container-image=${INSECURE_REGISTRY_URL}/google_containers/pause\:0.8.0"

@ -4,11 +4,6 @@
echo "configuring kubernetes (minion)"
if [ -z "$KUBE_NODE_IP" ]; then
# FIXME(yuanying): Set KUBE_NODE_IP correctly
KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
fi
CERT_DIR=/srv/kubernetes
PROTOCOL=https
FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
@ -52,8 +47,17 @@ sed -i '
# The hostname of the node is set to be the Nova name of the instance, and
# the option --hostname-override for kubelet uses the hostname to register the node.
# Using any other name will break the load balancer and cinder volume features.
HOSTNAME=$(hostname --short | sed 's/\.novalocal//')
KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 ${KUBE_CONFIG} --hostname-override=${HOSTNAME}"
HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 ${KUBE_CONFIG} --hostname-override=${HOSTNAME_OVERRIDE}"
if [ -n "$TRUST_ID" ]; then
KUBELET_ARGS="$KUBELET_ARGS --cloud-provider=openstack --cloud-config=/etc/sysconfig/kube_openstack_config"
fi
# Workaround for Cinder support (fixed in k8s >= 1.6)
if [ ! -f /usr/bin/udevadm ]; then
ln -s /sbin/udevadm /usr/bin/udevadm
fi
if [ -n "${INSECURE_REGISTRY_URL}" ]; then
KUBELET_ARGS="${KUBELET_ARGS} --pod-infra-container-image=${INSECURE_REGISTRY_URL}/google_containers/pause\:0.8.0"
@ -86,32 +90,6 @@ if [ "$NETWORK_DRIVER" = "flannel" ]; then
done
fi
if [ "$VOLUME_DRIVER" = "cinder" ]; then
CLOUD_CONFIG=/etc/kubernetes/kube_openstack_config
KUBERNETES=/etc/kubernetes
if [ ! -d ${KUBERNETES} -o ! -f ${CLOUD_CONFIG} ]; then
mkdir -p $KUBERNETES
fi
AUTH_URL=${AUTH_URL/v3/v2.0}
cat > $CLOUD_CONFIG <<EOF
[Global]
auth-url=$AUTH_URL
username=$USERNAME
password=$PASSWORD
region=$REGION_NAME
tenant-name=$TENANT_NAME
EOF
cat << _EOC_ >> /etc/kubernetes/kubelet
#KUBELET_ARGS="$KUBELET_ARGS --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config"
_EOC_
if [ ! -f /usr/bin/udevadm ]; then
ln -s /sbin/udevadm /usr/bin/udevadm
fi
fi
cat >> /etc/environment <<EOF
KUBERNETES_MASTER=$KUBE_MASTER_URI
EOF

@ -30,6 +30,7 @@ write_files:
TLS_DISABLED="$TLS_DISABLED"
CLUSTER_UUID="$CLUSTER_UUID"
MAGNUM_URL="$MAGNUM_URL"
VOLUME_DRIVER="$VOLUME_DRIVER"
HTTP_PROXY="$HTTP_PROXY"
HTTPS_PROXY="$HTTPS_PROXY"
NO_PROXY="$NO_PROXY"

@ -38,8 +38,6 @@ write_files:
WAIT_CURL="$WAIT_CURL"
KUBE_VERSION="$KUBE_VERSION"
TRUSTEE_USER_ID="$TRUSTEE_USER_ID"
TRUSTEE_USERNAME="$TRUSTEE_USERNAME"
TRUSTEE_PASSWORD="$TRUSTEE_PASSWORD"
TRUSTEE_DOMAIN_ID="$TRUSTEE_DOMAIN_ID"
TRUST_ID="$TRUST_ID"
INSECURE_REGISTRY_URL="$INSECURE_REGISTRY_URL"

@ -4,17 +4,14 @@
KUBE_OS_CLOUD_CONFIG=/etc/sysconfig/kube_openstack_config
# kubernetes backend only support keystone v2 at this point
AUTH_URL=$(echo "$AUTH_URL" | sed 's/v3/v2.0/')
# Generate a the configuration for Kubernetes services
# to talk to OpenStack Neutron
cat > $KUBE_OS_CLOUD_CONFIG <<EOF
[Global]
auth-url=$AUTH_URL
username=$USERNAME
password=$PASSWORD
tenant-name=$TENANT_NAME
user-id=$TRUSTEE_USER_ID
password=$TRUSTEE_PASSWORD
trust-id=$TRUST_ID
[LoadBalancer]
subnet-id=$CLUSTER_SUBNET
create-monitor=yes

@ -245,16 +245,9 @@ class BaseTemplateDefinition(TemplateDefinition):
extra_params['trustee_username'] = cluster.trustee_username
extra_params['trustee_password'] = cluster.trustee_password
# Only pass trust ID into the template when it is needed.
if (cluster_template.volume_driver == 'rexray' or
cluster_template.registry_enabled):
if CONF.trust.cluster_user_trust:
extra_params['trust_id'] = cluster.trust_id
else:
missing_setting = ('trust/cluster_user_trust = True')
msg = ('This cluster can only be created with %s in '
'magnum.conf')
raise exception.ConfigInvalid(msg % missing_setting)
# Only pass trust ID into the template if allowed by the config file
if CONF.trust.cluster_user_trust:
extra_params['trust_id'] = cluster.trust_id
else:
extra_params['trust_id'] = ""

@ -453,6 +453,7 @@ resources:
discovery_url: {get_param: discovery_url}
cluster_uuid: {get_param: cluster_uuid}
magnum_url: {get_param: magnum_url}
volume_driver: {get_param: volume_driver}
fixed_network: {get_attr: [network, fixed_network]}
fixed_subnet: {get_attr: [network, fixed_subnet]}
api_pool_id: {get_attr: [api_lb, pool_id]}

@ -48,6 +48,10 @@ parameters:
constraints:
- allowed_values: ["devicemapper", "overlay"]
volume_driver:
type: string
description: volume driver to use for container storage
flannel_network_cidr:
type: string
description: network range for flannel overlay network
@ -276,6 +280,7 @@ resources:
"$TLS_DISABLED": {get_param: tls_disabled}
"$CLUSTER_UUID": {get_param: cluster_uuid}
"$MAGNUM_URL": {get_param: magnum_url}
"$VOLUME_DRIVER": {get_param: volume_driver}
"$HTTP_PROXY": {get_param: http_proxy}
"$HTTPS_PROXY": {get_param: https_proxy}
"$NO_PROXY": {get_param: no_proxy}

@ -255,9 +255,7 @@ resources:
$NO_PROXY: {get_param: no_proxy}
$KUBE_VERSION: {get_param: kube_version}
$WAIT_CURL: {get_attr: [minion_wait_handle, curl_cli]}
$TRUSTEE_DOMAIN_ID: {get_param: trustee_domain_id}
$TRUSTEE_USER_ID: {get_param: trustee_user_id}
$TRUSTEE_USERNAME: {get_param: trustee_username}
$TRUSTEE_PASSWORD: {get_param: trustee_password}
$TRUST_ID: {get_param: trust_id}
$AUTH_URL: {get_param: auth_url}
@ -269,6 +267,12 @@ resources:
group: ungrouped
config: {get_file: ../../common/templates/kubernetes/fragments/write-kubeconfig.yaml}
write_kube_os_config:
type: OS::Heat::SoftwareConfig
properties:
group: ungrouped
config: {get_file: ../../common/templates/kubernetes/fragments/write-kube-os-config.sh}
make_cert:
type: OS::Heat::SoftwareConfig
properties:
@ -364,6 +368,7 @@ resources:
- config: {get_resource: disable_selinux}
- config: {get_resource: write_heat_params}
- config: {get_resource: write_kubeconfig}
- config: {get_resource: write_kube_os_config}
- config: {get_resource: make_cert}
- config: {get_resource: kube_examples}
- config: {get_resource: configure_docker_storage}

@ -286,7 +286,7 @@ class TestClusterConductorWithSwarm(base.TestCase):
'trustee_username': 'fake_trustee',
'trustee_password': 'fake_trustee_password',
'trustee_user_id': '7b489f04-b458-4541-8179-6a48a553e656',
'trust_id': '',
'trust_id': 'bd11efc5-d4e2-4dac-bbce-25e348ddf7de',
'auth_url': 'http://192.168.10.10:5000/v3',
'swarm_version': 'fake-version',
'swarm_strategy': u'spread',

Loading…
Cancel
Save