==================================================== Experimenting containerized VNFs with Kubernetes VIM ==================================================== In the past, Tacker only supports creating virtual machine based VNF using Heat. This section covers how to deploy `containerized VNF` using Kubernetes VIM in Tacker. Prepare Kubernetes VIM ====================== To use Kubernetes type of VNF, firstly user must register Kubernetes VIM. Tacker supports Kubernetes authentication with two types: basic authentication (username and password) or Bearer token. User can secure the connection to Kubernetes cluster by providing SSL certificate. The following ``vim-config.yaml`` file provides necessary information to register a Kubernetes VIM. .. code-block:: console auth_url: "https://192.168.11.110:6443" username: "admin" password: "admin" project_name: "default" ssl_ca_cert: None type: "kubernetes" More details about registering Kubernetes VIM, please refer [#first]_ Sample container TOSCA templates ================================ 1. One container per VDU example ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently, because Kubernetes does not support multiple networks such as choosing networks, connection points for applications, therefore users only deploys their applications with default networks (Pod and Service networks). In this case, user need to provide only information about VDU to create a VNF in Tacker. The following example shows TOSCA template of containerized VNF for pure Kubernetes environment with one container per VDU. **tosca-vnfd-containerized-two-containers.yaml** .. code-block:: yaml tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0 description: A sample containerized VNF with two containers per VDU metadata: template_name: sample-tosca-vnfd topology_template: node_templates: VDU1: type: tosca.nodes.nfv.VDU.Tacker properties: namespace: default mapping_ports: - "80:80" - "88:88" service_type: NodePort vnfcs: front_end: num_cpus: 0.5 mem_size: 512 MB image: nginx ports: - "80" rss_reader: num_cpus: 0.5 mem_size: 512 MB image: nickchase/rss-php-nginx:v1 ports: - "88" policies: - SP1: type: tosca.policies.tacker.Scaling targets: [VDU1] properties: min_instances: 1 max_instances: 3 target_cpu_utilization_percentage: 40 In "vnfcs", there are 2 components: front_end and rss_reader. We model them as Containers [#second]_ inside a Pod [#third]_. To provide recover ability of these containers, we put all of containers inside a Deployment [#fourth]_ object, that can warrant the number of replica with auto-healing automatically. The following table shows details about parameter of a Container. Config is translated to ConfigMap [#fifth]_, when user want to update the VNF (config), equivalent ConfigMap will be updated. .. code-block:: console +-----------------------------------------------------------------------------------------------+ | vnfcs | Example | Description | +-----------------------------------------------------------------------------------------------+ | name | front_end | Name of container | +-----------------------------------------------------------------------------------------------+ | num_cpus | 0.5 | Number of CPUs | +-----------------------------------------------------------------------------------------------+ | mem_size | 512 MB | Memory size | +-----------------------------------------------------------------------------------------------+ | image | nginx | Image to launch container | +-----------------------------------------------------------------------------------------------+ | ports | - "80" | Exposed ports in container | +-----------------------------------------------------------------------------------------------+ | command | ['/bin/sh','echo'] | Command when container was started | +-----------------------------------------------------------------------------------------------+ | args | ['hello'] | Args of command | +-----------------------------------------------------------------------------------------------+ | config | param0: key1 | Set variables | | | param1: key2 | | +-----------------------------------------------------------------------------------------------+ In Tacker, VDU is modeled as a Service [#sixth]_ in Kubernetes. Because Pods can be easily replaced by others, when the number of replica increased, workload should be shared between Pods. To do this task, we model VDU as Service, it acts as a Load balancer for Pods. Currently, we support some parameters as the following table. .. code-block:: console +--------------------------------------------------------------------------------------------------------------------------------+ | VDU properties | Example | Description | +--------------------------------------------------------------------------------------------------------------------------------+ | namespace | default | Namespace in Kubernetes where all objects are deployed | +--------------------------------------------------------------------------------------------------------------------------------+ | mapping_ports | - "443:443" | Published ports and target ports (container ports) of Service Kubernetes | | | - "80:8080" | | +--------------------------------------------------------------------------------------------------------------------------------+ | labels | "app: webserver" | Labels which is set for Kubernetes objects, it is used as Selector to | | | | Service can send requests to Pods | +--------------------------------------------------------------------------------------------------------------------------------+ | service_type | ClusterIP | Set service type for Service object. | | | | | +--------------------------------------------------------------------------------------------------------------------------------+ | vnfcs | | Vnfcs are modeled by Containers and Deployment object. User can limit | | | | resource, set image, publish container ports, set commands and variables | +--------------------------------------------------------------------------------------------------------------------------------+ User can also set scaling policy for VDU by adding the following policy. These information is translated to Horizontal Pod Autoscaler in Kubernetes. In the current scope, we just support auto-scaling with CPU utilization, more metrics will be added in the future. .. code-block:: yaml policies: - SP1: type: tosca.policies.tacker.Scaling targets: [VDU1] properties: min_instances: 1 max_instances: 3 target_cpu_utilization_percentage: 40 2. Two containers per VDU example ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Similar to the above example, in this scenario, we define 2 containers in VDU1. **tosca-vnfd-containerized.yaml** .. code-block:: yaml tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0 description: A sample containerized VNF with two containers per VDU metadata: template_name: sample-tosca-vnfd topology_template: node_templates: VDU1: type: tosca.nodes.nfv.VDU.Tacker properties: namespace: default mapping_ports: - "80:8080" labels: - "app: webserver" service_type: ClusterIP vnfcs: web_server: num_cpus: 0.5 mem_size: 512 MB image: celebdor/kuryr-demo ports: - "8080" config: | param0: key1 param1: key2 policies: - SP1: type: tosca.policies.tacker.Scaling targets: [VDU1] properties: min_instances: 1 max_instances: 3 target_cpu_utilization_percentage: 40 Viewing a containerized VNF ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create sample containerized VNF .. code-block:: console $ openstack vnf descriptor create --vnfd-file tosca-vnfd-containerized.yaml VNFD1 Created a new vnfd: +-----------------+-------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+-------------------------------------------------------------------------------------------------------+ | created_at | 2018-01-21 14:36:51.757044 | | description | A sample containerized VNF with one container per VDU | | id | fb4a0aa8-e410-4e73-abdc-d2808de155ef | | name | VNFD1 | | service_types | vnfd | | template_source | onboarded | | tenant_id | 2d22508be9694091bb2f03ce27911416 | | updated_at | | +-----------------+-------------------------------------------------------------------------------------------------------+ $ openstack vnf create --vnfd-name VNFD1 --vim-name vim-kubernetes VNF1 Created a new vnf: +----------------+-------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------+-------------------------------------------------------------------------------------------------------+ | created_at | 2018-01-21 14:37:23.318018 | | description | A sample containerized VNF with one container per VDU | | error_reason | | | id | 1faf776b-8d2b-4ee6-889d-e3b7c7310411 | | instance_id | default,svc-vdu1-05db44 | | mgmt_ip_address| | | name | VNF1 | | placement_attr | {"vim_name": "vim-kubernetes"} | | status | PENDING_CREATE | | tenant_id | 2d22508be9694091bb2f03ce27911416 | | updated_at | | | vim_id | 791830a6-45fd-468a-bd85-e07fe24e5ce3 | | vnfd_id | fb4a0aa8-e410-4e73-abdc-d2808de155ef | +----------------+-------------------------------------------------------------------------------------------------------+ $ openstack vnf list +--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+ | id | name | mgmt_ip_address | status | vim_id | vnfd_id | +--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+ | 1faf776b-8d2b-4ee6-889d-e3b7c7310411 | VNF1 | | ACTIVE | 791830a6-45fd-468a-bd85-e07fe24e5ce3 | fb4a0aa8-e410-4e73-abdc-d2808de155ef | +--------------------------------------+------+----------------------------+--------+--------------------------------------+--------------------------------------+ To test VNF is running in Kubernetes environment, we can check by running following commands .. code-block:: console $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 192.168.28.129 443/TCP 5h svc-vdu1-05db44 ClusterIP 192.168.28.187 80/TCP 12m $ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE svc-vdu1-05db44 1 1 1 1 16m $ kubectl get pod NAME READY STATUS RESTARTS AGE svc-vdu1-05db44-7dcb6b955d-wkh7d 1/1 Running 0 18m $ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE svc-vdu1-05db44 Deployment/svc-vdu1-05db44 / 40% 1 3 1 17m $ kubectl get configmap NAME DATA AGE svc-vdu1-05db44 2 17m User also can scale VNF manually, by running the following commands: .. code-block:: console $ openstack vnf scale --vnf-name VNF1 --scaling-policy-name SP1 --scaling-type out $ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE svc-vdu1-651815 2 2 2 1 3h $ kubectl get pods NAME READY STATUS RESTARTS AGE svc-vdu1-651815-5b894b8bfb-b6mzq 2/2 Running 0 3h svc-vdu1-651815-5b894b8bfb-b7f2c 2/2 Running 0 40s In the same way, user also scale in VNF with scaling-type is 'in'. The range of scaling manually is limited by 'min_instances' and 'max_instances' user provide in VNF template. Multi-Interface for C-VNF ========================= To use multi-interface for C-VNF, User should follow below procedure. 1. Checking kuryr.conf ~~~~~~~~~~~~~~~~~~~~~~ After installation, user should check kuryr.conf configuration. .. code-block:: console $ sudo cat /etc/kuryr/kuryr.conf | grep multi_vif_drivers multi_vif_drivers = npwg_multiple_interfaces 2. Adding K8s CustomResourceDefinition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To use CustomResourceDefinition, user needs to add CRD. User can make a additional network using yaml file like below. Create yaml file like below and register it. .. code-block:: console $ cat ./crdnetwork.yaml .. code-block:: yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: network-attachment-definitions.k8s.cni.cncf.io spec: group: k8s.cni.cncf.io version: v1 scope: Namespaced names: plural: network-attachment-definitions singular: network-attachment-definition kind: NetworkAttachmentDefinition shortNames: - net-attach-def validation: openAPIV3Schema: properties: spec: properties: config: type: string Register crdnetwork.yaml .. code-block:: console $ kubectl create -f ~/crdnetwork.yaml Get crd list .. code-block:: console $ kubectl get crd NAME CREATED AT kuryrnetpolicies.openstack.org 2019-07-31T02:23:54Z kuryrnets.openstack.org 2019-07-31T02:23:54Z network-attachment-definitions.k8s.cni.cncf.io 2019-07-31T02:23:55Z 3. Adding neutron subnet id information to k8s CRD ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To use neutron subnet in kubernetes, user should register neutron subnet to CRD. At first, user should create subnet.yaml file. .. code-block:: console $ cat ./kuryr-subnetname1.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: subnetname1 annotations: openstack.org/kuryr-config: '{"subnetId": "$subnet_id"}' After making a yaml file, user should create subnet with yaml file. .. code-block:: console $ kubectl create -f ~/kuryr-subnetname1.yaml After created, user can check subnet info like below. .. code-block:: console $ kubectl get net-attach-def NAME AGE k8s-multi-10 7d k8s-multi-11 7d Known Issues and Limitations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Does not support Volumes in Kubernetes - Horizontal Pod AutoScaler only support CPU utilization - Add support Kuryr-Kubernetes for making hybrid network in the future References ========== .. [#first] https://github.com/openstack/tacker/blob/master/doc/source/install/kubernetes_vim_installation.rst .. [#second] https://kubernetes.io/docs/concepts/workloads/pods/init-containers .. [#third] https://kubernetes.io/docs/concepts/workloads/pods/pod-overview .. [#fourth] https://kubernetes.io/docs/concepts/workloads/controllers/deployment .. [#fifth] https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap .. [#sixth] https://kubernetes.io/docs/concepts/services-networking/service