diff --git a/doc/source/install/developer/deploy-ovs-dpdk.rst b/doc/source/install/developer/deploy-ovs-dpdk.rst new file mode 100644 index 0000000000..82f6199dcd --- /dev/null +++ b/doc/source/install/developer/deploy-ovs-dpdk.rst @@ -0,0 +1,189 @@ +=============== +Deploy OVS-DPDK +=============== + +Requirements +============ + +A correct DPDK configuration depends heavily on the specific hardware resources +and its configuration. Before deploying Openvswitch with DPDK, check the amount +and type of available hugepages on the host OS. + +.. code-block:: shell + + cat /proc/meminfo | grep Huge + AnonHugePages: 0 kB + ShmemHugePages: 0 kB + HugePages_Total: 8 + HugePages_Free: 6 + HugePages_Rsvd: 0 + HugePages_Surp: 0 + Hugepagesize: 1048576 kB + +In this example, 8 hugepages of 1G size have been allocated. 2 of those are +being used and 6 are still available. + +More information on how to allocate and configure hugepages on the host OS can +be found in the `Openvswitch documentation +`_. + +In order to allow OVS inside a pod to make use of hugepages, the corresponding +type and amount of hugepages must be specified in the resource section of the +OVS chart's values.yaml: + +.. code-block:: yaml + + resources: + enabled: true + ovs: + db: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "1024Mi" + cpu: "2000m" + vswitchd: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "1024Mi" + cpu: "2000m" + # set resources to enabled and specify one of the following when using dpdk + hugepages-1Gi: "1Gi" + # hugepages-2Mi: "512Mi" + +Additionally, the default configuration of the neutron chart must be adapted according +to the underlying hardware. The corresponding configuration parameter is labeled with +"CHANGE-ME" in the script "values_overrides/dpdk.yaml". Specifically, the "ovs_dpdk" +configuration section should list all NICs which should be bound to DPDK with +their corresponding PCI-IDs. Moreover, the name of each NIC needs to be unique, +e.g., dpdk0, dpdk1, etc. + +.. code-block:: yaml + + network: + interface: + tunnel: br-phy + conf: + ovs_dpdk: + enabled: true + driver: uio_pci_generic + nics: + - name: dpdk0 + # CHANGE-ME: modify pci_id according to hardware + pci_id: '0000:05:00.0' + bridge: br-phy + migrate_ip: true + bridges: + - name: br-phy + bonds: [] + +In the example above, bonding isn't used and hence an empty list is passed in the "bonds" +section. + +Deployment +========== + +Once the above requirements are met, start deploying Openstack Helm using the deployment +scripts under the dpdk directory in an increasing order + +.. code-block:: shell + + ./tools/deployment/developer/dpdk/ + +One can also specify the name of Openstack release and container OS distribution as +overrides before running the deployment scripts, for instance, + +.. code-block:: shell + + export OPENSTACK_RELEASE=rocky + export CONTAINER_DISTRO_NAME=ubuntu + export CONTAINER_DISTRO_VERSION=bionic + +Note that OVS-DPDK deployment has been tested with Openstack Rocky release and Ubuntu +Bionic container distributions. If the above variables aren't set, the defaults (currently +Openstack Ocata and Ubuntu Xenial) will be used. + +Troubleshooting +=============== + +OVS startup failure +------------------- + +If OVS fails to start up because of no hugepages are available, check the +configuration of the OVS daemonset. Older versions of helm-toolkit were not +able to render hugepage configuration into the Kubernetes manifest and just +removed the hugepage attributes. If no hugepage configuration is defined for +the OVS daemonset, consider using a newer version of helm-toolkit. + +.. code-block:: shell + + kubectl get daemonset openvswitch-vswitchd -n openstack -o yaml + [...] + resources: + limits: + cpu: "2" + hugepages-1Gi: 1Gi + memory: 1Gi + requests: + cpu: 100m + memory: 128Mi + [...] + +Adding a DPDK port to Openvswitch fails +--------------------------------------- + +When adding a DPDK port (a NIC bound to DPDK) to OVS fails, one source of error +is related to an incorrect configuration with regards to the NUMA topology of +the underlying hardware. Every NIC is connected to one specific NUMA socket. In +order to use a NIC as DPDK port in OVS, the OVS configuration regarding +hugepages and PMD threas needs to match the NUMA topology. + +The NUMA socket a given NIC is connected to can be found in the ovs-vswitchd log: + +.. code-block:: + + kubectl logs -n openstack openvswitch-vswitchd-6h928 + [...] + 2019-07-02T13:42:06Z|00016|dpdk|INFO|EAL: PCI device 0000:00:04.0 on NUMA socket 1 + 2019-07-02T13:42:06Z|00018|dpdk|INFO|EAL: probe driver: 1af4:1000 net_virtio + [...] + +In this example, the NIC with PCI-ID 0000:00:04.0 is connected to NUMA socket +1. As a result, this NIC can only be used by OVS if + +1. hugepages have been allocated on NUMA socket 1 by OVS, and +2. PMD threads have been assigned to NUMA socket 1. + +To allocate hugepages to NUMA sockets in OVS, ensure that the +``socket_memory`` attribute in values.yaml specifies a value for the +corresponding NUMA socket. In the following example, OVS will use one 1G +hugepage for NUMA socket 0 and socket 1. + +.. code-block:: + + socket_memory: 1024,1024 + + +To allocate PMD threads to NUMA sockets in OVS, ensure that the ``pmd_cpu_mask`` +attribute in values.yaml includes CPU sockets on the corresponding NUMA socket. +In the example below, the mask of 0xf covers the first 4 CPU cores which are +dsitributed across NUMA sockets 0 and 1. + +.. code-block:: + + pmd_cpu_mask: 0xf + +The mapping of CPU cores to NUMA sockets can be determined by means of ``lspci``, for instance: + +.. code-block:: shell + + lspci | grep NUMA + NUMA node(s): 2 + NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 + NUMA node1 CPU(s): 1,3,5,7,9,11,13,15 + +More information can be found in the `Openvswitch documentation +`_. diff --git a/doc/source/install/developer/index.rst b/doc/source/install/developer/index.rst index 554bf050c0..ed4498a35e 100644 --- a/doc/source/install/developer/index.rst +++ b/doc/source/install/developer/index.rst @@ -10,5 +10,6 @@ Contents: kubernetes-and-common-setup deploy-with-nfs deploy-with-ceph + deploy-ovs-dpdk.rst exercise-the-cloud cleaning-deployment