Add external IP controller

Add ansible playbook to deploy external IP controller on k8s
cluster. It will deploy the application on specified nodes only
(the ones that have role "externalip" in ansible).

Change-Id: I85556d58db92a45968e49de86efd5f447bbcd086

doc/source/external_ip_controller.rst

@@ -0,0 +1,151 @@
+.. _external_ip_controller:
+
+=================================
+Installing External IP Controller
+=================================
+
+This document describes how to expose Kubernetes services using External IP
+Controller.
+
+Introduction
+~~~~~~~~~~~~
+
+One of the possible ways to expose k8s services on a bare metal deployment is
+using External IPs. Each node runs a kube-proxy process which programs
+iptables rules to trap access to External IPs and redirect them to the correct
+backends.
+
+So in order to accessa k8s service from the outside we just need to route public
+traffic to one of k8s worker nodes which has kube-proxy running and thus has
+needed iptables rules for External IPs.
+
+Deployment scheme
+~~~~~~~~~~~~~~~~~
+
+Description
+-----------
+
+External IP controller is a k8s application which is deployed on top of a k8s
+cluster and which configures External IPs on k8s worker node(s) to provide IP
+connectivity.
+
+For further details please read `External IP controller documentation
+<https://github.com/Mirantis/k8s-externalipcontroller/blob/master/doc/>`_
+
+Ansible Playbook
+----------------
+
+The playbook is ``utils/kargo/externalip.yaml`` and the ansible role is
+``utils/kargo/roles/externalip``.
+
+The nodes that have ``externalip`` role assigned to them will run External IP
+controller application which will manage Kubernetes services' external IPs.
+Playbook labels such nodes and then creates DaemonSet with appropriate
+``nodeSelector``.
+
+External IP scheduler will be running as a standard Deployment (ReplicaSet)
+with specified number of replicas.
+
+ECMP deployment
+~~~~~~~~~~~~~~~
+
+Deployment model
+----------------
+
+In this sample deployment we're going to deploy External IP controller on a set
+of nodes and provide load balancing and high availability for External IPs
+based on Equal-cost multi-path routing (ECMP).
+
+For further details please read `Documentation about ECMP deployment
+<https://github.com/Mirantis/k8s-externalipcontroller/blob/master/doc/ecmp-load-balancing.md>`_
+
+Inventory
+---------
+
+You can take inventory generated previously for Kargo deployment. Using
+``utils/kargo/externalip.yaml`` playbook with such inventory will deploy
+External IP controller on all ``kube-node`` worker nodes.
+
+Custom yaml
+-----------
+
+Custom Ansible yaml for ECMP deployment is stored here:
+``utils/jenkins/extip_ecmp.yaml``. Here is the content:
+
+::
+
+    # Type of deployment
+    extip_ctrl_app_kind: "DaemonSet"
+    # IP distribution model
+    extip_distribution: "all"
+    # Netmask for external IPs
+    extip_mask: 32
+    # Interface to bring IPs on, should be "lo" for ECMP
+    extip_iface: "lo"
+
+Deployment
+----------
+
+Just run the following ansible-playbook command:
+
+.. code:: sh
+
+      export ws=/home/workspace
+      /usr/bin/ansible-playbook -e ansible_ssh_pass=vagrant -u vagrant -b \
+      --become-user=root -i ${ws}/inventory/inventory.cfg \
+      -e @${ws}/utils/jenkins/extip_ecmp.yaml \
+      ${ws}/utils/kargo/externalip.yaml
+
+This will deploy the application according to your inventory.
+
+Routing
+-------
+
+This application only brings IPs up or down on a specified interface. We also
+need to provide routing to those nodes with external IPs. So for Kubernetes
+cluster with Calico networking plugin we already have ``calico-node`` container
+running on every k8s worker node. This container also includes BGP speaker
+which monitors local routing tables and announces changes via BGP protocol.
+So in order to include external IPs to BGP speaker export we need to add the
+following custom export filter for Calico:
+
+.. code:: sh
+
+      cat << EOF | etcdctl set /calico/bgp/v1/global/custom_filters/v4/lo_iface
+      if ( ifname = "lo" ) then {
+        if net != 127.0.0.0/8 then accept;
+      }
+      EOF
+
+Please note that this will only configure BGP for ``calico-node``. In order to
+announce routing to your network infrastructure you may want to peer Calico
+with routers. Please check this URL for details:
+
+`Kargo docs: Calico BGP Peering with border routers
+<https://github.com/kubernetes-incubator/kargo/blob/master/docs/calico.md#optional--bgp-peering-with-border-routers>`_
+
+Uninstalling and undoing customizations
+---------------------------------------
+
+Uninstall k8s applications by running the following commands on the first
+kube-master node in your ansible inventory:
+
+.. code:: sh
+
+      kubectl delete -f /etc/kubernetes/extip_scheduler.yml
+      kubectl delete -f /etc/kubernetes/extip_controller.yml
+
+Remove custom Calico export filter:
+
+.. code:: sh
+
+      etcdctl rm /calico/bgp/v1/global/custom_filters/v4/lo_iface
+
+Also remove external IPs from `lo` interface on the nodes with the command
+like this:
+
+.. code:: sh
+
+      ip ad del 10.0.0.7/32 dev lo
+
+Where ``10.0.0.7/32`` is external IP.

doc/source/index.rst

@@ -22,6 +22,7 @@ Contents
    collect_info
    specify_hyperkube_image
    configurable_params
+   external_ip_controller
 
 Search in this guide
 ~~~~~~~~~~~~~~~~~~~~

utils/jenkins/extip_ecmp.yaml

@@ -0,0 +1,6 @@
+# External IP playbook settings to provide ECMP
+extip_image_tag: "release-0.2.1"
+extip_ctrl_app_kind: "DaemonSet"
+extip_distribution: "all"
+extip_mask: 32
+extip_iface: "lo"

utils/kargo/externalip.yaml

@@ -0,0 +1,4 @@
+---
+- hosts: kube-node
+  roles:
+    - { role: externalip }

utils/kargo/roles/externalip/defaults/main.yml

@@ -0,0 +1,54 @@
+---
+# All nodes will be labeled and then label will be used as "nodeSelector" for
+# external IP controller application. You can spacify this label name here.
+extip_node_label: "externalip"
+
+# Image params
+extip_image_repo: "mirantis/k8s-externalipcontroller"
+extip_image_tag: "release-0.2.1"
+
+# If multiple external IPs claim controllers are running we need to
+# define how to distribute IPs.
+# Valid values are:
+#   "balance" - IPs are balanced across controllers
+#   "single" - only one controller will hold all IPs. Not yet supported in
+#              multiple controllers mode. If you need to run on one node please
+#              use extip_ctrl_app_kind=Deployment and extip_ctrl_replicas=1
+#   "all" - all controllers will bring up all IPs (for ECMP, for example)
+extip_distribution: "balance"
+
+# External IPs network mask
+extip_mask: 24
+
+# Interface to bring external IPs on
+extip_iface: "eth0"
+
+# Kubernetes namespace for the application
+k8s_namespace: "default"
+
+#####
+# K8s controller app params
+extip_ctrl_app: "claimcontroller"
+# App kind, valid values are: "Deployment", "DaemonSet"
+extip_ctrl_app_kind: "Deployment"
+extip_ctrl_replicas: 1
+extip_ctrl_label: "externalipcontroller"
+extip_ctrl_image_pull_policy: "IfNotPresent"
+# Verbosity
+extip_ctrl_verbose: 5
+# Heartbeat
+extip_ctrl_hb: "500ms"
+extip_ctrl_hostname: "{{ ansible_hostname }}"
+
+#####
+# K8s scheduler app params
+extip_sched_app: "claimscheduler"
+extip_sched_label: "claimscheduler"
+extip_sched_image_pull_policy: "IfNotPresent"
+extip_sched_replicas: 2
+# Verbosity
+extip_sched_verbose: 5
+# Scheduler leader elect, string ("true" or "false")
+extip_sched_leader_elect: "true"
+# Monitor
+extip_sched_monitor: "1s"

utils/kargo/roles/externalip/tasks/main.yml

@@ -0,0 +1,65 @@
+---
+- name: ExtIP Controller | Get pods
+  shell: "kubectl get pods -o wide"
+  run_once: true
+  register: pods
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Get list of nodes with labels
+  shell: "kubectl get node --show-labels"
+  run_once: true
+  register: k8s_nodes
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Get list of nodes with IPs
+  shell: 'kubectl get nodes -o jsonpath=''{range .items[*]}{@.metadata.name}{" "}{range @.status.addresses[?(@.type == "InternalIP")]}{@.address}{"\n"}{end}{end}'''
+  register: k8s_name_cmd
+  run_once: true
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Set fact with node-ip search pattern
+  set_fact:
+    k8s_ip_pattern: ".* {{ ip }}$"
+
+- name: ExtIP Controller | Find k8s node name by IP address
+  set_fact:
+    k8s_name: "{{ (k8s_name_cmd.stdout_lines | select('match', k8s_ip_pattern) | join(',')).split(' ')[0] }}"
+
+- name: ExtIP Controller | Print k8s node names
+  debug:
+    msg: "{{ k8s_name }}"
+
+- name: ExtIP Controller | Set fact with node-label search pattern
+  set_fact:
+    k8s_label_pattern: "^{{ k8s_name }} .*[,\ ]{{ extip_node_label }}=true.*"
+
+- name: ExtIP Controller | Find matches for node by label
+  set_fact:
+    matches: "{{ k8s_nodes.stdout_lines | select('match', k8s_label_pattern) | list }}"
+
+- name: ExtIP Controller | Label node if needed
+  shell: "kubectl label nodes {{ k8s_name }} {{ extip_node_label }}=true"
+  when: "{{ (matches | length) < 1 }}"
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Upload claimcontroller config
+  run_once: true
+  template: src=controller.yaml.j2 dest=/etc/kubernetes/extip_controller.yml
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Upload claimscheduler config
+  run_once: true
+  template: src=scheduler.yaml.j2 dest=/etc/kubernetes/extip_scheduler.yml
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Create claimcontroller
+  run_once: true
+  shell: "kubectl create -f /etc/kubernetes/extip_controller.yml"
+  when: pods.stdout.find("{{ extip_ctrl_app }}-") == -1
+  delegate_to: "{{groups['kube-master'][0]}}"
+
+- name: ExtIP Controller | Create claimscheduler
+  run_once: true
+  shell: "kubectl create -f /etc/kubernetes/extip_scheduler.yml"
+  when: pods.stdout.find("{{ extip_sched_app }}-") == -1
+  delegate_to: "{{groups['kube-master'][0]}}"

utils/kargo/roles/externalip/templates/controller.yaml.j2

@@ -0,0 +1,31 @@
+apiVersion: extensions/v1beta1
+kind: {{ extip_ctrl_app_kind }}
+metadata:
+  name: {{ extip_ctrl_app }}
+spec:
+{% if extip_ctrl_app_kind != "DaemonSet" %}
+  replicas: {{ extip_ctrl_replicas }}
+{% endif %}
+  template:
+    metadata:
+      labels:
+        app: {{ extip_ctrl_label }}
+    spec:
+      hostNetwork: true
+      nodeSelector:
+        {{ extip_node_label }}: "true"
+      containers:
+      - name: externalipcontroller
+        image: {{ extip_image_repo }}:{{ extip_image_tag }}
+        imagePullPolicy: {{ extip_ctrl_image_pull_policy }}
+        securityContext:
+          privileged: true
+        command:
+          - ipmanager
+          - claimcontroller
+          - --iface={{ extip_iface }}
+          - --logtostderr
+          - --v={{ extip_ctrl_verbose }}
+          - --hb={{ extip_ctrl_hb }}
+          - --hostname={% if extip_distribution == "all" %}ecmp{% else %}{{ extip_ctrl_hostname }}
+{% endif %}

utils/kargo/roles/externalip/templates/scheduler.yaml.j2

@@ -0,0 +1,23 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: {{ extip_sched_app }}
+spec:
+  replicas: {{ extip_sched_replicas }}
+  template:
+    metadata:
+      labels:
+        app: {{ extip_sched_label }}
+    spec:
+      containers:
+        - name: externalipcontroller
+          image: {{ extip_image_repo }}:{{ extip_image_tag }}
+          imagePullPolicy: {{ extip_sched_image_pull_policy }}
+          command:
+            - ipmanager
+            - scheduler
+            - --mask={{ extip_mask }}
+            - --logtostderr
+            - --v={{ extip_sched_verbose }}
+            - --leader-elect={{ extip_sched_leader_elect }}
+            - --monitor={{ extip_sched_monitor }}