Browse Source

Fix usage of the trustee user in K8S Cinder plugin

Closes-Bug: #1672667
Change-Id: I702818777ea4664ecd560c4b7a02431c86988e17
Mathieu Velten 2 years ago
parent
commit
73f4d639c5

+ 4
- 51
doc/source/dev/kubernetes-load-balancer.rst View File

@@ -78,57 +78,7 @@ neutron_lbaas.conf::
78 78
 
79 79
 To configure LBaaS v1 or v2, refer to the Neutron documentation.
80 80
 
81
-To enable the load balancer, log into each master node of your cluster and
82
-perform the following steps:
83
-
84
-1. Configure kube-apiserver::
85
-
86
-    sudo vi /etc/kubernetes/apiserver
87
-
88
-   Comment out the line::
89
-
90
-    #KUBE_API_ARGS="--runtime_config=api/all=true"
91
-
92
-   Uncomment the line::
93
-
94
-    KUBE_API_ARGS="--runtime_config=api/all=true --cloud_config=/etc/sysconfig/kube_openstack_config --cloud_provider=openstack"""
95
-
96
-2. Configure kube-controller-manager::
97
-
98
-    sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
99
-
100
-   Immediately after the lines::
101
-
102
-    - controller-manager
103
-    - --master=http://127.0.0.1:8080
104
-    - --service-account-private-key-file=/etc/kubernetes/ssl/server.key
105
-    - --root-ca-file=/etc/kubernetes/ssl/ca.crt
106
-
107
-   Add the following lines::
108
-
109
-    - --cloud_config=/etc/sysconfig/kube_openstack_config
110
-    - --cloud_provider=openstack
111
-
112
-   When the file is saved, the pod will automatically restart the
113
-   kube-controller-manager container to pick up the change.
114
-
115
-3. Enter OpenStack user credential::
116
-
117
-    sudo vi /etc/sysconfig/kube_openstack_config
118
-
119
-   The username and tenant-name entries have been filled in with the
120
-   Keystone values of the user who created the cluster.  Enter the password
121
-   of this user on the entry for password::
122
-
123
-    password=ChangeMe
124
-
125
-4. Restart the Kubernetes API server::
126
-
127
-    sudo service kube-apiserver restart
128
-    service kube-apiserver status
129
-
130
-This only needs to be done once.  The steps can be reversed to disable the
131
-load balancer feature. Before deleting the Kubernetes cluster, make sure to
81
+Before deleting the Kubernetes cluster, make sure to
132 82
 delete all the services that created load balancers. Because the Neutron
133 83
 objects created by Kubernetes are not managed by Heat, they will not be
134 84
 deleted by Heat and this will cause the cluster-delete operation to fail. If
@@ -138,6 +88,9 @@ lb-healthmonitor) and then run cluster-delete again.
138 88
 Steps for the users
139 89
 ===================
140 90
 
91
+This feature requires the OpenStack cloud provider to be enabled.
92
+To do so, enable the cinder support (--volume-driver cinder).
93
+
141 94
 For the user, publishing the service endpoint externally involves the following
142 95
 2 steps:
143 96
 

+ 2
- 49
doc/source/userguide.rst View File

@@ -2202,12 +2202,8 @@ Following are some examples for using Cinder as persistent storage.
2202 2202
 Using Cinder in Kubernetes
2203 2203
 ++++++++++++++++++++++++++
2204 2204
 
2205
-**NOTE:** This feature requires Kubernetes version 1.1.1 or above and
2206
-Docker version 1.8.3 or above.  The public Fedora image from Atomic
2207
-currently meets this requirement.
2208
-
2209
-**NOTE:** The following steps are a temporary workaround, and Magnum's
2210
-development team is working on a long term solution to automate these steps.
2205
+**NOTE:** This feature requires Kubernetes version 1.5.0 or above.
2206
+The public Fedora image from Atomic currently meets this requirement.
2211 2207
 
2212 2208
 1. Create the ClusterTemplate.
2213 2209
 
@@ -2230,49 +2226,6 @@ development team is working on a long term solution to automate these steps.
2230 2226
                           --cluster-template k8s-cluster-template \
2231 2227
                           --node-count 1
2232 2228
 
2233
-
2234
-3. Configure kubelet.
2235
-
2236
-   To allow Kubernetes to interface with Cinder, log into each minion
2237
-   node of your cluster and perform step 4 through 6::
2238
-
2239
-    sudo vi /etc/kubernetes/kubelet
2240
-
2241
-   Comment out the line::
2242
-
2243
-    #KUBELET_ARGS=--config=/etc/kubernetes/manifests --cadvisor-port=4194
2244
-
2245
-   Uncomment the line::
2246
-
2247
-    #KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config"
2248
-
2249
-
2250
-4. Enter OpenStack user credential::
2251
-
2252
-    sudo vi /etc/kubernetes/kube_openstack_config
2253
-
2254
-  The username, tenant-name and region entries have been filled in with the
2255
-  Keystone values of the user who created the cluster.  Enter the password
2256
-  of this user on the entry for password::
2257
-
2258
-    password=ChangeMe
2259
-
2260
-5. Restart Kubernetes services::
2261
-
2262
-    sudo systemctl restart kubelet
2263
-
2264
-   On restart, the new configuration enables the Kubernetes cloud provider
2265
-   plugin for OpenStack, along with the necessary credential for kubelet
2266
-   to authenticate with Keystone and to make request to OpenStack services.
2267
-
2268
-6. Install nsenter::
2269
-
2270
-    sudo docker run -v /usr/local/bin:/target jpetazzo/nsenter
2271
-
2272
-   The nsenter utility is used by Kubernetes to run new processes within
2273
-   existing kernel namespaces. This allows the kubelet agent to manage storage
2274
-   for pods.
2275
-
2276 2229
 Kubernetes is now ready to use Cinder for persistent storage.
2277 2230
 Following is an example illustrating how Cinder is used in a pod.
2278 2231
 

+ 11
- 13
magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh View File

@@ -4,11 +4,6 @@
4 4
 
5 5
 echo "configuring kubernetes (master)"
6 6
 
7
-if [ -z "$KUBE_NODE_IP" ]; then
8
-    # FIXME(yuanying): Set KUBE_NODE_IP correctly
9
-    KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
10
-fi
11
-
12 7
 sed -i '
13 8
     /^KUBE_ALLOW_PRIV=/ s/=.*/="--allow-privileged='"$KUBE_ALLOW_PRIV"'"/
14 9
 ' /etc/kubernetes/config
@@ -30,6 +25,10 @@ if [ -n "${ADMISSION_CONTROL_LIST}" ] && [ "${TLS_DISABLED}" == "False" ]; then
30 25
     KUBE_ADMISSION_CONTROL="--admission-control=${ADMISSION_CONTROL_LIST}"
31 26
 fi
32 27
 
28
+if [ -n "$TRUST_ID" ]; then
29
+    KUBE_API_ARGS="$KUBE_API_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
30
+fi
31
+
33 32
 sed -i '
34 33
     /^KUBE_API_ADDRESS=/ s/=.*/="'"${KUBE_API_ADDRESS}"'"/
35 34
     /^KUBE_SERVICE_ADDRESSES=/ s|=.*|="--service-cluster-ip-range='"$PORTAL_NETWORK_CIDR"'"|
@@ -38,10 +37,7 @@ sed -i '
38 37
     /^KUBE_ADMISSION_CONTROL=/ s/=.*/="'"${KUBE_ADMISSION_CONTROL}"'"/
39 38
 ' /etc/kubernetes/apiserver
40 39
 cat << _EOC_ >> /etc/kubernetes/apiserver
41
-#Uncomment the following line to disable Load Balancer feature
42 40
 KUBE_API_ARGS="$KUBE_API_ARGS"
43
-#Uncomment the following line to enable Load Balancer feature
44
-#KUBE_API_ARGS="$KUBE_API_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
45 41
 _EOC_
46 42
 
47 43
 # Add controller manager args
@@ -49,16 +45,18 @@ KUBE_CONTROLLER_MANAGER_ARGS=""
49 45
 if [ -n "${ADMISSION_CONTROL_LIST}" ] && [ "${TLS_DISABLED}" == "False" ]; then
50 46
     KUBE_CONTROLLER_MANAGER_ARGS="--service-account-private-key-file=/srv/kubernetes/server.key --root-ca-file=/srv/kubernetes/ca.crt"
51 47
 fi
48
+
49
+if [ -n "$TRUST_ID" ]; then
50
+    KUBE_CONTROLLER_MANAGER_ARGS="$KUBE_CONTROLLER_MANAGER_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
51
+fi
52
+
52 53
 sed -i '
53 54
     /^KUBELET_ADDRESSES=/ s/=.*/="--machines='""'"/
54 55
     /^KUBE_CONTROLLER_MANAGER_ARGS=/ s#\(KUBE_CONTROLLER_MANAGER_ARGS\).*#\1="'"${KUBE_CONTROLLER_MANAGER_ARGS}"'"#
55 56
 ' /etc/kubernetes/controller-manager
56
-cat << _EOC_ >> /etc/kubernetes/controller-manager
57
-#Uncomment the following line to enable Kubernetes Load Balancer feature
58
-#KUBE_CONTROLLER_MANAGER_ARGS="\$KUBE_CONTROLLER_MANAGER_ARGS --cloud-config=/etc/sysconfig/kube_openstack_config --cloud-provider=openstack"
59
-_EOC_
60 57
 
61
-KUBELET_ARGS="--register-node=true --register-schedulable=false --config=/etc/kubernetes/manifests --hostname-override=$KUBE_NODE_IP"
58
+HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
59
+KUBELET_ARGS="--register-node=true --register-schedulable=false --config=/etc/kubernetes/manifests --hostname-override=${HOSTNAME_OVERRIDE}"
62 60
 
63 61
 if [ -n "${INSECURE_REGISTRY_URL}" ]; then
64 62
     KUBELET_ARGS="${KUBELET_ARGS} --pod-infra-container-image=${INSECURE_REGISTRY_URL}/google_containers/pause\:0.8.0"

+ 11
- 33
magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh View File

@@ -4,11 +4,6 @@
4 4
 
5 5
 echo "configuring kubernetes (minion)"
6 6
 
7
-if [ -z "$KUBE_NODE_IP" ]; then
8
-    # FIXME(yuanying): Set KUBE_NODE_IP correctly
9
-    KUBE_NODE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
10
-fi
11
-
12 7
 CERT_DIR=/srv/kubernetes
13 8
 PROTOCOL=https
14 9
 FLANNEL_OPTIONS="-etcd-cafile $CERT_DIR/ca.crt \
@@ -52,8 +47,17 @@ sed -i '
52 47
 # The hostname of the node is set to be the Nova name of the instance, and
53 48
 # the option --hostname-override for kubelet uses the hostname to register the node.
54 49
 # Using any other name will break the load balancer and cinder volume features.
55
-HOSTNAME=$(hostname --short | sed 's/\.novalocal//')
56
-KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 ${KUBE_CONFIG} --hostname-override=${HOSTNAME}"
50
+HOSTNAME_OVERRIDE=$(hostname --short | sed 's/\.novalocal//')
51
+KUBELET_ARGS="--config=/etc/kubernetes/manifests --cadvisor-port=4194 ${KUBE_CONFIG} --hostname-override=${HOSTNAME_OVERRIDE}"
52
+
53
+if [ -n "$TRUST_ID" ]; then
54
+    KUBELET_ARGS="$KUBELET_ARGS --cloud-provider=openstack --cloud-config=/etc/sysconfig/kube_openstack_config"
55
+fi
56
+
57
+# Workaround for Cinder support (fixed in k8s >= 1.6)
58
+if [ ! -f /usr/bin/udevadm ]; then
59
+    ln -s /sbin/udevadm /usr/bin/udevadm
60
+fi
57 61
 
58 62
 if [ -n "${INSECURE_REGISTRY_URL}" ]; then
59 63
     KUBELET_ARGS="${KUBELET_ARGS} --pod-infra-container-image=${INSECURE_REGISTRY_URL}/google_containers/pause\:0.8.0"
@@ -86,32 +90,6 @@ if [ "$NETWORK_DRIVER" = "flannel" ]; then
86 90
     done
87 91
 fi
88 92
 
89
-if [ "$VOLUME_DRIVER" = "cinder" ]; then
90
-    CLOUD_CONFIG=/etc/kubernetes/kube_openstack_config
91
-    KUBERNETES=/etc/kubernetes
92
-    if [ ! -d ${KUBERNETES} -o ! -f ${CLOUD_CONFIG} ]; then
93
-        mkdir -p $KUBERNETES
94
-    fi
95
-    AUTH_URL=${AUTH_URL/v3/v2.0}
96
-cat > $CLOUD_CONFIG <<EOF
97
-[Global]
98
-auth-url=$AUTH_URL
99
-username=$USERNAME
100
-password=$PASSWORD
101
-region=$REGION_NAME
102
-tenant-name=$TENANT_NAME
103
-EOF
104
-
105
-cat << _EOC_ >> /etc/kubernetes/kubelet
106
-#KUBELET_ARGS="$KUBELET_ARGS --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config"
107
-_EOC_
108
-
109
-    if [ ! -f /usr/bin/udevadm ]; then
110
-        ln -s /sbin/udevadm /usr/bin/udevadm
111
-    fi
112
-
113
-fi
114
-
115 93
 cat >> /etc/environment <<EOF
116 94
 KUBERNETES_MASTER=$KUBE_MASTER_URI
117 95
 EOF

+ 1
- 0
magnum/drivers/common/templates/kubernetes/fragments/write-heat-params-master.yaml View File

@@ -29,6 +29,7 @@ write_files:
29 29
       TLS_DISABLED="$TLS_DISABLED"
30 30
       CLUSTER_UUID="$CLUSTER_UUID"
31 31
       MAGNUM_URL="$MAGNUM_URL"
32
+      VOLUME_DRIVER="$VOLUME_DRIVER"
32 33
       HTTP_PROXY="$HTTP_PROXY"
33 34
       HTTPS_PROXY="$HTTPS_PROXY"
34 35
       NO_PROXY="$NO_PROXY"

+ 0
- 2
magnum/drivers/common/templates/kubernetes/fragments/write-heat-params.yaml View File

@@ -37,8 +37,6 @@ write_files:
37 37
       WAIT_CURL="$WAIT_CURL"
38 38
       KUBE_VERSION="$KUBE_VERSION"
39 39
       TRUSTEE_USER_ID="$TRUSTEE_USER_ID"
40
-      TRUSTEE_USERNAME="$TRUSTEE_USERNAME"
41 40
       TRUSTEE_PASSWORD="$TRUSTEE_PASSWORD"
42
-      TRUSTEE_DOMAIN_ID="$TRUSTEE_DOMAIN_ID"
43 41
       TRUST_ID="$TRUST_ID"
44 42
       INSECURE_REGISTRY_URL="$INSECURE_REGISTRY_URL"

+ 3
- 6
magnum/drivers/common/templates/kubernetes/fragments/write-kube-os-config.sh View File

@@ -4,17 +4,14 @@
4 4
 
5 5
 KUBE_OS_CLOUD_CONFIG=/etc/sysconfig/kube_openstack_config
6 6
 
7
-# kubernetes backend only support keystone v2 at this point
8
-AUTH_URL=$(echo "$AUTH_URL" | sed 's/v3/v2.0/')
9
-
10 7
 # Generate a the configuration for Kubernetes services
11 8
 # to talk to OpenStack Neutron
12 9
 cat > $KUBE_OS_CLOUD_CONFIG <<EOF
13 10
 [Global]
14 11
 auth-url=$AUTH_URL
15
-username=$USERNAME
16
-password=$PASSWORD
17
-tenant-name=$TENANT_NAME
12
+user-id=$TRUSTEE_USER_ID
13
+password=$TRUSTEE_PASSWORD
14
+trust-id=$TRUST_ID
18 15
 [LoadBalancer]
19 16
 subnet-id=$CLUSTER_SUBNET
20 17
 create-monitor=yes

+ 3
- 11
magnum/drivers/heat/template_def.py View File

@@ -21,7 +21,6 @@ import six
21 21
 from magnum.common import clients
22 22
 from magnum.common import exception
23 23
 import magnum.conf
24
-from magnum.i18n import _LE
25 24
 from magnum.i18n import _LW
26 25
 
27 26
 from requests import exceptions as req_exceptions
@@ -247,16 +246,9 @@ class BaseTemplateDefinition(TemplateDefinition):
247 246
         extra_params['trustee_username'] = cluster.trustee_username
248 247
         extra_params['trustee_password'] = cluster.trustee_password
249 248
 
250
-        # Only pass trust ID into the template when it is needed.
251
-        if (cluster_template.volume_driver == 'rexray' or
252
-                cluster_template.registry_enabled):
253
-            if CONF.trust.cluster_user_trust:
254
-                extra_params['trust_id'] = cluster.trust_id
255
-            else:
256
-                missing_setting = ('trust/cluster_user_trust = True')
257
-                msg = _LE('This cluster can only be created with %s in '
258
-                          'magnum.conf')
259
-                raise exception.ConfigInvalid(msg % missing_setting)
249
+        # Only pass trust ID into the template if allowed by the config file
250
+        if CONF.trust.cluster_user_trust:
251
+            extra_params['trust_id'] = cluster.trust_id
260 252
         else:
261 253
             extra_params['trust_id'] = ""
262 254
 

+ 1
- 0
magnum/drivers/k8s_fedora_atomic_v1/templates/kubecluster.yaml View File

@@ -438,6 +438,7 @@ resources:
438 438
           discovery_url: {get_param: discovery_url}
439 439
           cluster_uuid: {get_param: cluster_uuid}
440 440
           magnum_url: {get_param: magnum_url}
441
+          volume_driver: {get_param: volume_driver}
441 442
           fixed_network: {get_attr: [network, fixed_network]}
442 443
           fixed_subnet: {get_attr: [network, fixed_subnet]}
443 444
           api_pool_id: {get_attr: [api_lb, pool_id]}

+ 5
- 0
magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml View File

@@ -48,6 +48,10 @@ parameters:
48 48
     constraints:
49 49
       - allowed_values: ["devicemapper", "overlay"]
50 50
 
51
+  volume_driver:
52
+    type: string
53
+    description: volume driver to use for container storage
54
+
51 55
   flannel_network_cidr:
52 56
     type: string
53 57
     description: network range for flannel overlay network
@@ -264,6 +268,7 @@ resources:
264 268
             "$TLS_DISABLED": {get_param: tls_disabled}
265 269
             "$CLUSTER_UUID": {get_param: cluster_uuid}
266 270
             "$MAGNUM_URL": {get_param: magnum_url}
271
+            "$VOLUME_DRIVER": {get_param: volume_driver}
267 272
             "$HTTP_PROXY": {get_param: http_proxy}
268 273
             "$HTTPS_PROXY": {get_param: https_proxy}
269 274
             "$NO_PROXY": {get_param: no_proxy}

+ 7
- 2
magnum/drivers/k8s_fedora_atomic_v1/templates/kubeminion.yaml View File

@@ -249,9 +249,7 @@ resources:
249 249
             $NO_PROXY: {get_param: no_proxy}
250 250
             $KUBE_VERSION: {get_param: kube_version}
251 251
             $WAIT_CURL: {get_attr: [minion_wait_handle, curl_cli]}
252
-            $TRUSTEE_DOMAIN_ID: {get_param: trustee_domain_id}
253 252
             $TRUSTEE_USER_ID: {get_param: trustee_user_id}
254
-            $TRUSTEE_USERNAME: {get_param: trustee_username}
255 253
             $TRUSTEE_PASSWORD: {get_param: trustee_password}
256 254
             $TRUST_ID: {get_param: trust_id}
257 255
             $AUTH_URL: {get_param: auth_url}
@@ -263,6 +261,12 @@ resources:
263 261
       group: ungrouped
264 262
       config: {get_file: ../../common/templates/kubernetes/fragments/write-kubeconfig.yaml}
265 263
 
264
+  write_kube_os_config:
265
+    type: OS::Heat::SoftwareConfig
266
+    properties:
267
+      group: ungrouped
268
+      config: {get_file: ../../common/templates/kubernetes/fragments/write-kube-os-config.sh}
269
+
266 270
   make_cert:
267 271
     type: OS::Heat::SoftwareConfig
268 272
     properties:
@@ -352,6 +356,7 @@ resources:
352 356
         - config: {get_resource: disable_selinux}
353 357
         - config: {get_resource: write_heat_params}
354 358
         - config: {get_resource: write_kubeconfig}
359
+        - config: {get_resource: write_kube_os_config}
355 360
         - config: {get_resource: make_cert}
356 361
         - config: {get_resource: kube_examples}
357 362
         - config: {get_resource: configure_docker_storage}

+ 1
- 1
magnum/tests/unit/conductor/handlers/test_swarm_cluster_conductor.py View File

@@ -286,7 +286,7 @@ class TestClusterConductorWithSwarm(base.TestCase):
286 286
             'trustee_username': 'fake_trustee',
287 287
             'trustee_password': 'fake_trustee_password',
288 288
             'trustee_user_id': '7b489f04-b458-4541-8179-6a48a553e656',
289
-            'trust_id': '',
289
+            'trust_id': 'bd11efc5-d4e2-4dac-bbce-25e348ddf7de',
290 290
             'auth_url': 'http://192.168.10.10:5000/v3',
291 291
             'swarm_version': 'fake-version',
292 292
             'swarm_strategy': u'spread',

+ 1
- 1
specs/containers-service.rst View File

@@ -175,7 +175,7 @@ with the Containers Service, and can be controlled only by a Nova virt driver.
175 175
  |           +-------+ |  | +-----+                   |
176 176
  |                     |  |                           |
177 177
  +-----------+---------+  +---------------+-----------+
178
-             |                            |            
178
+             |                            |
179 179
  +-----------+----+ Compute Host ---------|-----------+
180 180
  |                                    +---+---+       |
181 181
  |                               +----+ Relay +---+   |

Loading…
Cancel
Save