================================================ Bare Metal Deployment Guide for kolla-kubernetes ================================================ .. warning:: This documentation is under construction and some browsers do not update on changes to docs.openstack.org. To resolve this problem, click refresh on the browser. The docs do work perfectly if followed but If you still run into trouble, please join #openstack-kolla and we can learn together how to solve whatever issues faced. Likely others in the community face the same issues. .. note:: This document was tested against CentOS 7.3 and Ubuntu 16.04 Host OS and AIO environments. All the steps should be run as non-root user. If you follow this guide as the root user, helm cannot be found in ``/usr/local/bin/`` because the path ``/usr/local/bin`` is not defaulted to enabled in CentOS 7. Introduction ============ There are many ways to deploy Kubernetes. This guide has been tested only with kubeadm. The documentation for kubeadm is here: https://kubernetes.io/docs/getting-started-guides/kubeadm/ Here is a video shown at a Kubernetes specific meetup on February 15th, 2017: https://youtu.be/rHCCUP2odd8 There are two steps to deploying kolla-kubernetes. The first step involves deploying Kubernetes. The second step involves deploying Kolla that is compatible with Kubernetes. Host machine requirements ========================= The host machine must satisfy the following minimum requirements: - 2 network interfaces - 8GB main memory - 40GB disk space Dependencies:: docker == 1.12.6 helm >= 2.4.1 kubectl >= 1.8.0 kubeadm >= 1.8.0 kubelet >= 1.8.0 kubernetes-cni >= 0.5.1 .. note:: When working with Kubernetes it is considered a useful practice to open a unique terminal window and run the command that watches all Kubernetes's processes. This operation will show changes as they occur within Kubernetes. This is referred to as the `watch terminal` in this documentation:: watch -d kubectl get pods --all-namespaces .. note:: Alternatively run this which will provide more information including pod ip addresses, but needs a wider terminal as a result:: watch -d kubectl get pods --all-namespaces -o wide Step 1: Deploy Kubernetes ========================= .. note:: This document recommends Kubernetes 1.6.4 or later. Note that kubernetes 1.6.3 is DOA and will not work. 1.7.0 appears to be stable and is working in the gates, and is the current latest version. .. warning:: This documentation assumes a POD CIDR of 10.1.0.0/16 and a service CIDR of 10.3.3.0/24. Two rules must be followed when reading this guide. 1. The service and pod cidr cannot overlap 2. The address spaces cannot already be allocated by your organization If the POD and CIDR addresses overlap in this documentation with your organizations's IP address ranges, they may be changed. Simply substitute anywhere these addresses are used with the custom cidrs you have chosen. .. note:: If you fail to turn off SELinux and firewalld, Kubernetes will fail. Turn off SELinux:: sudo setenforce 0 sudo sed -i 's/enforcing/permissive/g' /etc/selinux/config Turn off firewalld:: sudo systemctl stop firewalld sudo systemctl disable firewalld .. note:: This operation configures the Kubernetes YUM repository. This step only needs to be done once per server or VM. .. warning:: gpgcheck=0 is set below because the currently signed RPMs don't match the yum-key.gpg key distributed by Kubernetes. YMMV. CentOS ------ Write the Kubernetes repository file:: sudo tee /etc/yum.repos.d/kubernetes.repo< kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF sudo cp -aR kubernetes.list /etc/apt/sources.list.d/kubernetes.list sudo apt-get update Install Kubernetes 1.6.4 or later and other dependencies:: sudo apt-get install -y docker.io kubelet kubeadm kubectl kubernetes-cni Centos and Ubuntu ----------------- Enable and start Docker:: sudo systemctl enable docker sudo systemctl start docker Ubuntu ------ Enable the proper CGROUP driver:: CGROUP_DRIVER=$(sudo docker info | grep "Cgroup Driver" | awk '{print $3}') sudo sed -i "s|KUBELET_KUBECONFIG_ARGS=|KUBELET_KUBECONFIG_ARGS=--cgroup-driver=$CGROUP_DRIVER |g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Centos and Ubuntu ----------------- Setup the DNS server with the service CIDR:: sudo sed -i 's/10.96.0.10/10.3.3.10/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf .. note:: Kubernetes uses x.x.x.10 as the DNS server. The Kolla developers don't know precisely why this is the case, however, current speculation is that that 1..9 are reserved for future expansion of Kubernetes infrastructure services. Reload the hand-modified service files:: sudo systemctl daemon-reload Stop kubelet if it is running:: sudo systemctl stop kubelet Enable and start docker and kubelet:: sudo systemctl enable kubelet sudo systemctl start kubelet Deploy Kubernetes with kubeadm:: sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.3.0/24 .. note:: pod-network-cidr is a network private to Kubernetes that the PODs within Kubernetes communicate on. The service-cidr is where IP addresses for Kubernetes services are allocated. There is no recommendation that the pod network should be /16 network in upstream documentation however, the Kolla developers have found through experience that each node consumes an entire /24 network, so this configuration would permit 255 Kubernetes nodes. .. note:: If the following issue occurs after running this command: `[preflight] Some fatal errors occurred: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1` There are two work-arounds: - Add `net.bridge.bridge-nf-call-ip6tables = 1` and `net.bridge.bridge-nf-call-iptables = 1` to ``/etc/sysctl.conf`` - Type `sysctl -p` to apply the settings from /etc/sysctl.conf - Type `sysctl net.bridge.bridge-nf-call-ip6tables` and `sysctl net.bridge.bridge-nf-call-iptables` to verify the values are set to 1. - Or alternatively Run with `--skip-preflight-checks`. This runs the risk of missing other issues that may be flagged. Load the kubedm credentials into the system:: mkdir -p $HOME/.kube sudo -H cp /etc/kubernetes/admin.conf $HOME/.kube/config sudo -H chown $(id -u):$(id -g) $HOME/.kube/config .. note:: Until this step is done, the `watch terminal` will not return information. The CNI driver is the networking driver that Kubernetes uses. Kolla uses Canal currently in the gate and tests with it hundreds of times per day via extensive gating mechanisms. Kolla recommends the use of Canal although other CNI drivers may be used if they are properly configured. Deploy the Canal CNI driver:: curl -L https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/1.6/rbac.yaml -o rbac.yaml kubectl apply -f rbac.yaml curl -L https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/1.6/canal.yaml -o canal.yaml sed -i "s@10.244.0.0/16@10.1.0.0/16@" canal.yaml kubectl apply -f canal.yaml Finally untaint the node (mark the master node as schedulable) so that PODs can be scheduled to this AIO deployment:: kubectl taint nodes --all=true node-role.kubernetes.io/master:NoSchedule- .. note:: Canal CNI appears to untaint the master node in later versions. This is a reported bug: https://github.com/projectcalico/canal/issues/77 Kubernetes must start completely before verification will function properly. In your `watch terminal`, confirm that Kubernetes has completed initialization by observing that the dns pod is in `3/3 Running` state. If you fail to wait, Step 2 will fail. Step 2: Validate Kubernetes =========================== After executing Step 2, a working Kubernetes deployment should be achieved. Launch a busybox container:: kubectl run -i -t $(uuidgen) --image=busybox --restart=Never Verify DNS works properly by running below command within the busybox container:: nslookup kubernetes This should return a nslookup result without error:: $ kubectl run -i -t $(uuidgen) --image=busybox --restart=Never Waiting for pod default/33c30c3b-8130-408a-b32f-83172bca19d0 to be running, status is Pending, pod ready: false # nslookup kubernetes Server: 10.3.3.10 Address 1: 10.3.3.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.3.3.1 kubernetes.default.svc.cluster.local .. warning:: If nslookup kubernetes fails, kolla-kubernetes will not deploy correctly. If this occurs check that all preceding steps have been applied correctly, and that the range of IP addresses chosen make sense to your particular environment. Running in a VM can cause nested virtualization and or performance issues. If still stuck seek further assistance from the Kubernetes or Kolla communities. Step 3: Deploying kolla-kubernetes ================================== Override default RBAC settings:: kubectl update -f <(cat < get_helm.sh chmod 700 get_helm.sh ./get_helm.sh helm init .. note:: In your `watch terminal` wait for the tiller pod to successfully come up. Verify both the client and server version of Helm are consistent:: helm version Install repositories necessary to install packaging:: sudo yum install -y epel-release ansible python-pip python-devel .. note:: You may find it helpful to create a directory to contain the files downloaded during the installation of kolla-kubernetes. To do that:: mkdir kolla-bringup cd kolla-bringup Clone kolla-ansible:: git clone http://github.com/openstack/kolla-ansible Clone kolla-kubernetes:: git clone http://github.com/openstack/kolla-kubernetes Install kolla-ansible and kolla-kubernetes:: sudo pip install -U kolla-ansible/ kolla-kubernetes/ Copy default Kolla configuration to /etc:: sudo cp -aR /usr/share/kolla-ansible/etc_examples/kolla /etc Copy default kolla-kubernetes configuration to /etc:: sudo cp -aR kolla-kubernetes/etc/kolla-kubernetes /etc Generate default passwords via SPRNG:: sudo kolla-kubernetes-genpwd Create a Kubernetes namespace to isolate this Kolla deployment:: kubectl create namespace kolla Label the AIO node as the compute and controller node:: kubectl label node $(hostname) kolla_compute=true kubectl label node $(hostname) kolla_controller=true .. warning: The kolla-kubernetes deliverable has two configuration files. This is a little clunky and we know about the problem :) We are working on getting all configuration into cloud.yaml. Until that is fixed the variable in globals.yml `kolla_install_type` must have the same contents as the variable in cloud.yaml `install_type`. In this document we use the setting `source` although `binary` could also be used. Modify Kolla ``/etc/kolla/globals.yml`` configuration file:: 1. Set `network_interface` in `/etc/kolla/globals.yml` to the Management interface name. E.g: `eth0`. 2. Set `neutron_external_interface` in `/etc/kolla/globals.yml` to the Neutron interface name. E.g: `eth1`. This is the external interface that Neutron will use. It must not have an IP address assigned to it. Add required configuration to the end of ``/etc/kolla/globals.yml``:: cat < add-to-globals.yml kolla_install_type: "source" tempest_image_alt_id: "{{ tempest_image_id }}" tempest_flavor_ref_alt_id: "{{ tempest_flavor_ref_id }}" neutron_plugin_agent: "openvswitch" api_interface_address: 0.0.0.0 tunnel_interface_address: 0.0.0.0 orchestration_engine: KUBERNETES memcached_servers: "memcached" keystone_admin_url: "http://keystone-admin:35357/v3" keystone_internal_url: "http://keystone-internal:5000/v3" keystone_public_url: "http://keystone-public:5000/v3" glance_registry_host: "glance-registry" neutron_host: "neutron" keystone_database_address: "mariadb" glance_database_address: "mariadb" nova_database_address: "mariadb" nova_api_database_address: "mariadb" neutron_database_address: "mariadb" cinder_database_address: "mariadb" ironic_database_address: "mariadb" placement_database_address: "mariadb" rabbitmq_servers: "rabbitmq" openstack_logging_debug: "True" enable_heat: "no" enable_cinder: "yes" enable_cinder_backend_lvm: "yes" enable_cinder_backend_iscsi: "yes" enable_cinder_backend_rbd: "no" enable_ceph: "no" enable_elasticsearch: "no" enable_kibana: "no" glance_backend_ceph: "no" cinder_backend_ceph: "no" nova_backend_ceph: "no" EOF cat ./add-to-globals.yml | sudo tee -a /etc/kolla/globals.yml For operators using virtualization for evaluation purposes please enable QEMU libvirt functionality and enable a workaround for a bug in libvirt:: sudo mkdir /etc/kolla/config sudo tee /etc/kolla/config/nova.conf< 150:: ls | grep ".tgz" | wc -l Create a local cloud.yaml file for the deployment of the charts:: cat < cloud.yaml global: kolla: all: docker_registry: docker.io image_tag: "4.0.0" kube_logger: false external_vip: "192.168.7.105" base_distro: "centos" install_type: "source" tunnel_interface: "docker0" keystone: all: admin_port_external: "true" dns_name: "192.168.7.105" public: all: port_external: "true" rabbitmq: all: cookie: 67 glance: api: all: port_external: "true" cinder: api: all: port_external: "true" volume_lvm: all: element_name: cinder-volume daemonset: lvm_backends: - '192.168.7.105': 'cinder-volumes' ironic: conductor: daemonset: selector_key: "kolla_conductor" nova: placement_api: all: port_external: true novncproxy: all: port: 6080 port_external: true openvswitch: all: add_port: true ext_bridge_name: br-ex ext_interface_name: enp1s0f1 setup_bridge: true horizon: all: port_external: true EOF .. warning:: This file is populated with several values that will need to be customized to your environment, this is explained below. .. note:: The placement api is enabled by default. If you wish to disable the placement API to run Mitaka or Newton images, this can be done by setting the `variable global.kolla.nova.all.placement_api_enabled` to `false` in the cloud.yaml file. .. note:: The default docker registry is ``docker.io``. If you want to use local registry, modify the value of ``docker_registry`` to your local registry .. note:: The next operations are not a simple copy and paste as the rest of this document is structured. In `/etc/kolla/globals.yml` you assigned your Management interface name to `network_interface` (E.g. `eth0`) - we will refer to this as: `YOUR_NETWORK_INTERFACE_NAME_FROM_GLOBALS.YML`. Record the ip address assigned to `YOUR_NETWORK_INTERFACE_NAME_FROM_GLOBALS.YML` (E.g. `10.240.43.81`). We will refer to this as: `YOUR_NETWORK_INTERFACE_ADDRESS_FROM_GLOBALS.YML`. Also record the name of the `neutron_external_interface` from `/etc/kolla/globals.yml` (E.g. `eth1`). We will refer to this as: `YOUR_NEUTRON_INTERFACE_NAME_FROM_GLOBALS.YML`. Replace all occurrences of `192.168.7.105` with `YOUR_NETWORK_INTERFACE_ADDRESS_FROM_GLOBALS.YML`:: sed -i "s@192.168.7.105@YOUR_NETWORK_INTERFACE_ADDRESS_FROM_GLOBALS.YML@g" ./cloud.yaml .. note:: This operation will have changed the values set in: `external_vip`, `dns_name` and `cinder-volumes` variables. Replace `enp1s0f1` with `YOUR_NEUTRON_INTERFACE_NAME_FROM_GLOBALS.YML`:: sed -i "s@enp1s0f1@YOUR_NEUTRON_INTERFACE_NAME_FROM_GLOBALS.YML@g" ./cloud.yaml .. note:: This operation will have changed the value set in: `ext_interface_name` variable. Replace `docker0` with the management interface name (E.g. `eth0`) used for connectivity between nodes in kubernetes cluster, in most cases it is `YOUR_NETWORK_INTERFACE_NAME_FROM_GLOBALS.YML`:: sed -i "s@docker0@YOUR_NETWORK_INTERFACE_NAME_FROM_GLOBALS.YML@g" ./cloud.yaml .. note:: This operation will have changed the value set in: `tunnel_interface` variable. Start mariadb first and wait for it to enter into Running state:: helm install --debug kolla-kubernetes/helm/service/mariadb --namespace kolla --name mariadb --values ./cloud.yaml Start many of the remaining service level charts:: helm install --debug kolla-kubernetes/helm/service/rabbitmq --namespace kolla --name rabbitmq --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/memcached --namespace kolla --name memcached --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/keystone --namespace kolla --name keystone --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/glance --namespace kolla --name glance --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/cinder-control --namespace kolla --name cinder-control --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/horizon --namespace kolla --name horizon --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/openvswitch --namespace kolla --name openvswitch --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/neutron --namespace kolla --name neutron --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/nova-control --namespace kolla --name nova-control --values ./cloud.yaml helm install --debug kolla-kubernetes/helm/service/nova-compute --namespace kolla --name nova-compute --values ./cloud.yaml Deploy iSCSI support with Cinder LVM (Optional) The Cinder LVM implementation requires a volume group to be set up. This can either be a real physical volume or a loopback mounted file for development. Use ``pvcreate`` and ``vgcreate`` to create the volume group. For example with the devices ``/dev/sdb`` and ``/dev/sdc``:: pvcreate /dev/sdb /dev/sdc vgcreate cinder-volumes /dev/sdb /dev/sdc During development, it may be desirable to use file backed block storage. It is possible to use a file and mount it as a block device via the loopback system:: mknod /dev/loop2 b 7 2 dd if=/dev/zero of=/var/lib/cinder_data.img bs=1G count=20 losetup /dev/loop2 /var/lib/cinder_data.img pvcreate /dev/loop2 vgcreate cinder-volumes /dev/loop2 Note that in the event where iSCSI daemon is active on the host, there is a need to perform the following steps before executing the cinder-volume-lvm Helm chart to avoid the iscsd container from going into crash loops:: sudo systemctl stop iscsid sudo systemctl stop iscsid.socket Execute the cinder-volume-lvm Helm chart:: helm install --debug kolla-kubernetes/helm/service/cinder-volume-lvm --namespace kolla --name cinder-volume-lvm --values ./cloud.yaml In the `watch terminal` wait for all pods to enter into Running state. If you didn't run watch in a different terminal, you can run it now:: watch -d kubectl get pods --all-namespaces Generate openrc file:: kolla-kubernetes/tools/build_local_admin_keystonerc.sh ext source ~/keystonerc_admin .. note:: The ``ext`` option to create the keystonerc creates a keystonerc file that is compatible with this guide. Install OpenStack clients:: sudo pip install "python-openstackclient" sudo pip install "python-neutronclient" sudo pip install "python-cinderclient" Bootstrap the cloud environment and create a VM as requested:: kolla-ansible/tools/init-runonce Create a floating IP address and add to the VM:: openstack server add floating ip demo1 $(openstack floating ip create public1 -f value -c floating_ip_address) Troubleshooting and Tear Down ============================= TroubleShooting --------------- .. note:: This is just a list of popular commands the community has suggested they use a lot. This is by no means a comprehensive guide to debugging kubernetes or kolla. Determine IP and port information:: $ kubectl get svc -n kube-system NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE canal-etcd 10.3.3.100 6666/TCP 16h kube-dns 10.3.3.10 53/UDP,53/TCP 16h tiller-deploy 10.3.3.7 44134/TCP 16h $ kubectl get svc -n kolla NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE cinder-api 10.3.3.6 10.240.43.81 8776/TCP 15h glance-api 10.3.3.150 10.240.43.81 9292/TCP 15h glance-registry 10.3.3.119 9191/TCP 15h horizon 10.3.3.15 10.240.43.81 80/TCP 15h keystone-admin 10.3.3.253 10.240.43.81 35357/TCP 15h keystone-internal 10.3.3.155 5000/TCP 15h keystone-public 10.3.3.214 10.240.43.81 5000/TCP 15h mariadb 10.3.3.57 3306/TCP 15h memcached 10.3.3.180 11211/TCP 15h neutron-server 10.3.3.145 10.240.43.81 9696/TCP 15h nova-api 10.3.3.96 10.240.43.81 8774/TCP 15h nova-metadata 10.3.3.118 8775/TCP 15h nova-novncproxy 10.3.3.167 10.240.43.81 6080/TCP 15h nova-placement-api 10.3.3.192 10.240.43.81 8780/TCP 15h rabbitmq 10.3.3.158 5672/TCP 15h rabbitmq-mgmt 10.3.3.105 15672/TCP 15h View all k8's namespaces:: $ kubectl get namespaces NAME STATUS AGE default Active 16h kolla Active 15h kube-public Active 16h kube-system Active 16h Kolla Describe a pod in full detail:: kubectl describe pod ceph-admin -n kolla ... View all deployed services:: $ kubectl get deployment -n kube-system NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kube-dns 1 1 1 1 20h tiller-deploy 1 1 1 1 20h View configuration maps:: $ kubectl get configmap -n kube-system NAME DATA AGE canal-config 4 20h cinder-control.v1 1 20h extension-apiserver-authentication 6 20h glance.v1 1 20h horizon.v1 1 20h keystone.v1 1 20h kube-proxy 1 20h mariadb.v1 1 20h memcached.v1 1 20h neutron.v1 1 20h nova-api-create.v1 1 19h nova-cell0-create-db-job.v1 1 19h nova-compute.v1 1 19h nova-control.v1 1 19h openvswitch.v1 1 20h rabbitmq.v1 1 20h General Cluster information:: $ kubectl cluster-info Kubernetes master is running at https://192.168.122.2:6443 KubeDNS is running at https://192.168.122.2:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns View all jobs:: $ kubectl get jobs --all-namespaces NAMESPACE NAME DESIRED SUCCESSFUL AGE kolla cinder-create-db 1 1 20h kolla cinder-create-keystone-endpoint-admin 1 1 20h kolla cinder-create-keystone-endpoint-adminv2 1 1 20h kolla cinder-create-keystone-endpoint-internal 1 1 20h kolla cinder-create-keystone-endpoint-internalv2 1 1 20h kolla cinder-create-keystone-endpoint-public 1 1 20h View all deployments:: $ kubectl get deployments --all-namespaces NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kolla cinder-api 1 1 1 1 20h kolla glance-api 1 1 1 1 20h kolla glance-registry 1 1 1 1 20h kolla horizon 1 1 1 1 20h kolla keystone 1 1 1 1 20h kolla memcached 1 1 1 1 20h kolla neutron-server 1 1 1 1 20h kolla nova-api 1 1 1 1 20h kolla nova-novncproxy 1 1 1 1 20h kolla placement-api 1 1 1 1 20h kube-system kube-dns 1 1 1 1 20h kube-system tiller-deploy 1 1 1 1 20h View secrets:: $ kubectl get secrets NAME TYPE DATA AGE default-token-3dzfp kubernetes.io/service-account-token 3 20h View docker images:: $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE gcr.io/kubernetes-helm/tiller v2.3.1 38527daf791d 7 days ago 56 MB quay.io/calico/cni v1.6.2 db2dedf2181a 2 weeks ago 65.08 MB gcr.io/google_containers/kube-proxy-amd64 v1.6.0 746d1460005f 3 weeks ago 109.2 MB ... Tear Down --------- .. warning:: Some of these steps are dangerous. Be warned. To cleanup the database entry for a specific service such as nova:: helm install --debug /opt/kolla-kubernetes/helm/service/nova-cleanup --namespace kolla --name nova-cleanup --values cloud.yaml To delete a Helm release:: helm delete mariadb --purge To delete all Helm releases:: helm delete mariadb --purge helm delete rabbitmq --purge helm delete memcached --purge helm delete keystone --purge helm delete glance --purge helm delete cinder-control --purge helm delete horizon --purge helm delete openvswitch --purge helm delete neutron --purge helm delete nova-control --purge helm delete nova-compute --purge helm delete nova-cell0-create-db-job --purge helm delete cinder-volume-lvm --purge To clean up the host volumes between runs:: sudo rm -rf /var/lib/kolla/volumes/* To clean up Kubernetes and all docker containers entirely, run this command, reboot, and run these commands again:: sudo kubeadm reset Other cleanups if your environment is corrup that may be useful:: sudo rm -rf /etc/kolla sudo rm -rf /etc/kubernetes sudo rm -rf /etc/kolla-kubernetes Using OpenStack =============== If you were able to successfully reach the end of this guide and `demo1` was successfully deployed, here is a fun list of things you can do with your new cluster. Access Horizon GUI ------------------ 1. Determine Horizon `EXTERNAL IP` Address:: $ kubectl get svc horizon --namespace=kolla NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE horizon 10.3.3.237 10.240.43.175 80/TCP 1d 2. Determine username and password from keystone:: $ cat ~/keystonerc_admin | grep OS_USERNAME export OS_USERNAME=admin $ cat ~/keystonerc_admin | grep OS_PASSWORD export OS_PASSWORD=Sr6XMFXvbvxQCJ3Cib1xb0gZ3lOtBOD8FCxOcodU 3. Run a browser that has access to your network, and access Horizon GUI with the `EXTERNAL IP` from Step 1, using the credentials from Step 2.