From 419391e97d195459914aa764850cd99c40708be5 Mon Sep 17 00:00:00 2001 From: Dmitrii Shcherbakov Date: Wed, 2 Feb 2022 17:54:08 +0300 Subject: [PATCH] Document remote-managed port usage considerations While most of the SR-IOV related documentation resides in the Neutron repository which is going to have a separate section on the topic of supporting remote-managed ports and off-path networking backends, there are still some things specific to Nova which are worth documenting in Nova docs. https://docs.openstack.org/neutron/latest/admin/config-sriov.html Implements: blueprint integration-with-off-path-network-backends Change-Id: I3c5fe8ec0539e10d07b1b4888e9833bc7ede1d04 --- doc/source/admin/networking.rst | 36 ++++++++++++++++ doc/source/admin/pci-passthrough.rst | 64 ++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+) diff --git a/doc/source/admin/networking.rst b/doc/source/admin/networking.rst index 61f23f9804cc..9005232cc371 100644 --- a/doc/source/admin/networking.rst +++ b/doc/source/admin/networking.rst @@ -37,6 +37,42 @@ A full guide on configuring and using SR-IOV is provided in the Nova will ignore PCI devices reported by the hypervisor if the address is outside of these ranges. +.. versionadded:: 25.0.0 + + For information on creating servers with remotely-managed SR-IOV network + interfaces of SmartNIC DPUs, refer to the relevant section in + :neutron-doc:`Networking Guide `. + + **Limitations** + + * Only VFs are supported and they must be tagged in the Nova Compute + configuration in the ``passthrough_whitelist`` option as + ``remote_managed: "true"``. There is no auto-discovery of this based + on vendor and product IDs; + * Either VF or its respective PF must expose a PCI VPD capability with a + unique card serial number according to the PCI/PCIe specifications + (see `the Libvirt docs `_ to + get an example of how VPD data is represented and what to expect). If + this is not the case, those devices will not appear in allocation pools; + * Only the Libvirt driver is capable of supporting this feature at the + time of writing; + * The support for VPD capability handling in Libvirt was added in release + `7.9.0 `_ - older + versions are not supported by this feature; + * All compute nodes must be upgraded to the Yoga release in order for + scheduling of nodes with ``VNIC_TYPE_REMOTE_MANAGED`` ports to succeed; + * The same limitations apply to operations like live migration as with + `legacy SR-IOV ports `_; + * Clearing a VLAN by programming VLAN 0 must not result in errors in the + VF kernel driver at the compute host. Before v8.1.0 Libvirt clears + a VLAN by programming VLAN 0 before passing a VF through to the guest + which may result in an error depending on your driver and kernel version + (see, for example, `this bug `_ + which discusses a case relevant to one driver). As of Libvirt v8.1.0, + EPERM errors encountered while programming VLAN 0 are ignored if + VLAN clearning is not explicitly requested in the device XML (i.e. + VLAN 0 is not specified explicitly). + NUMA Affinity ------------- diff --git a/doc/source/admin/pci-passthrough.rst b/doc/source/admin/pci-passthrough.rst index 727a63070de4..5f131b325a0c 100644 --- a/doc/source/admin/pci-passthrough.rst +++ b/doc/source/admin/pci-passthrough.rst @@ -156,6 +156,70 @@ Refer to :oslo.config:option:`pci.alias` for syntax information. Once configured, restart the :program:`nova-compute` service. +Special Tags +^^^^^^^^^^^^ + +When specified in :oslo.config:option:`pci.passthrough_whitelist` some tags +have special meaning: + +``physical_network`` + Associates a device with a physical network label which corresponds to the + ``physical_network`` attribute of a network segment object in Neutron. For + virtual networks such as overlays a value of ``null`` should be specified + as follows: ``"physical_network": null``. In the case of physical networks, + this tag is used to supply the metadata necessary for identifying a switched + fabric to which a PCI device belongs and associate the port with the correct + network segment in the networking backend. Besides typical SR-IOV scenarios, + this tag can be used for remote-managed devices in conjunction with the + ``remote_managed`` tag. + +``remote_managed`` + Used to specify whether a PCI device is managed remotely or not. By default, + devices are implicitly tagged as ``"remote_managed": "false"`` but and they + must be tagged as ``"remote_managed": "true"`` if ports with + ``VNIC_TYPE_REMOTE_MANAGED`` are intended to be used. Once that is done, + those PCI devices will not be available for allocation for regular + PCI passthrough use. Specifying ``"remote_managed": "true"`` is only valid + for SR-IOV VFs and specifying it for PFs is prohibited. + + .. important:: + It is recommended that PCI VFs that are meant to be remote-managed + (e.g. the ones provided by SmartNIC DPUs) are tagged as remote-managed in + order to prevent them from being allocated for regular PCI passthrough since + they have to be programmed accordingly at the host that has access to the + NIC switch control plane. If this is not done, instances requesting regular + SR-IOV ports may get a device that will not be configured correctly and + will not be usable for sending network traffic. + + .. important:: + For the Libvirt virt driver, clearing a VLAN by programming VLAN 0 must not + result in errors in the VF kernel driver at the compute host. Before v8.1.0 + Libvirt clears a VLAN before passing a VF through to the guest which may + result in an error depending on your driver and kernel version (see, for + example, `this bug `_ + which discusses a case relevant to one driver). As of Libvirt v8.1.0, EPERM + errors encountered while programming a VLAN are ignored if VLAN clearning is + not explicitly requested in the device XML. + +``trusted`` + If a port is requested to be trusted by specifying an extra option during + port creation via ``--binding-profile trusted=true``, only devices tagged as + ``trusted: "true"`` will be allocated to instances. Nova will then configure + those devices as trusted by the network controller through its PF device driver. + The specific set of features allowed by the trusted mode of a VF will differ + depending on the network controller itself, its firmware version and what a PF + device driver version allows to pass to the NIC. Common features to be affected + by this tag are changing the VF MAC address, enabling promiscuous mode or + multicast promiscuous mode. + + .. important:: + While the ``trusted tag`` does not directly conflict with the + ``remote_managed`` tag, network controllers in SmartNIC DPUs may prohibit + setting the ``trusted`` mode on a VF via a PF device driver in the first + place. It is recommended to test specific devices, drivers and firmware + versions before assuming this feature can be used. + + Configure ``nova-scheduler`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~