0c9c60d5ca
Closes-Bug: #1932154 Co-Authored-By: Dmitrii Shcherbakov <dmitrii.shcherbakov@canonical.com> Change-Id: I5b5e7957cfe8020001777fd40e038eaafb5fb894
206 lines
10 KiB
ReStructuredText
206 lines
10 KiB
ReStructuredText
.. _smartnic_dpu:
|
|
|
|
===============================
|
|
Off-path SmartNIC DPUs with OVN
|
|
===============================
|
|
|
|
The purpose of this page is to describe how off-path SmartNIC DPU hardware
|
|
can be integrated with Neutron when OVN mechanism driver is used. For an
|
|
in-depth discussion of underlying mechanisms it is recommended to get
|
|
familiar with the following specifications
|
|
|
|
* Neutron `Off-path SmartNIC DPU Port Binding with OVN specification`_;
|
|
|
|
* `Nova Integration With Off-path Network Backends specification`_.
|
|
|
|
Overview
|
|
--------
|
|
|
|
A class of devices collectively referred to as off-path SmartNIC DPUs introduces
|
|
an important change to earlier architectures where compute and networking agents
|
|
used to coexist at the hypervisor host: networking control plane components
|
|
are now moved to the SmartNIC DPU's CPU side which includes ``ovs-vswitchd``
|
|
and ``ovn-controller``. The following diagram provides an overview of the
|
|
components involved::
|
|
|
|
┌────────────────────────────────────┐
|
|
│ Hypervisor │ LoM Ports
|
|
│ ┌───────────┐ ┌───────────┐ │ (on-board,
|
|
│ │ Instance │ │ Nova │ ├──┐ optional)
|
|
│ │(e.g. QEMU)│ │ Compute │ │ ├─────────┐
|
|
│ │ │ │ │ ├──┘ │
|
|
│ └───────────┘ └───────────┘ │ │
|
|
│ │ │
|
|
└────────────────┬─┬───────┬─┬──┬────┘ │
|
|
│ │ │ │ │ │
|
|
│ │ │ │ │ Control Traffic │
|
|
Instance VF │ │ │ │ │ PF associated │
|
|
│ │ │ │ │ with an uplink │
|
|
│ │ │ │ │ port or a VF. │
|
|
│ │ │ │ │ (used to replace│
|
|
│ │ │ │ │ LoM) │
|
|
┌────────────────────────────────────┼─┼───────┼─┼──┼─┐ │
|
|
│ SmartNIC DPU Board │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ ┌──────────────┐ Control traffic │ │ │ │ │ │ │
|
|
│ │ App. CPU │ via PFs or VFs ┌┴─┴───────┴─┴┐ │ │ │
|
|
│ ├──────────────┤ (DC Fabric) │ │ │ │ │
|
|
│ │ovn-controller├─────────────────┼─┐ │ │ │ │
|
|
│ ├──────────────┤ │ │ │ │ │ │
|
|
│ │ovs-vswitchd │ │ │NIC Switch │ │ │ │
|
|
│ ├──────────────┤ │ │ASIC/FPGA │ │ │ │
|
|
│ │ Neutron OVN │ │ │ │ │ │ │
|
|
│ │metadata agent│ │ │ │ │ │ │
|
|
│ ├──────────────┤Port representors│ │ │ │ │ │
|
|
│ │ br-int ├═════════════════┤ │ │ │ │ │
|
|
│ └──────────────┘ └─┼───┬─┬─────┘ │ │ │
|
|
┌ ┴─ ─ ─ ┐Optional port for │ │ │ │ │ │
|
|
─┤OOB Port initial NIC switch config │ │ │uplink │ │ │
|
|
└ ┬─ ─ ─ ┘ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │
|
|
└─────────────────────────────────────┼───┼─┼───────┼─┘ │
|
|
│ │ │ │ │
|
|
┌──┼───┴─┴───────┼────────┐ │
|
|
│ │ │ │ │
|
|
│ │ DC Fabric ├────────┼────────┘
|
|
│ │ │ │
|
|
└──┼─────────────┼────────┘
|
|
│ │
|
|
│ ┌───┴──────┐
|
|
│ │ │
|
|
┌───▼──┐ ┌───▼───┐ ┌────▼────┐
|
|
│OVN SB│ │Neutron│ │Placement│
|
|
└──────┘ │Server │ │ │
|
|
└───────┘ └─────────┘
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
* OpenStack Yoga or newer;
|
|
|
|
* `Open vSwitch`_ >= 2.17;
|
|
|
|
* `Open Virtual Network`_ >= 21.12.0;
|
|
|
|
* `OVN VIF`_ >= 21.12.0;
|
|
|
|
* A SmartNIC DPU with the following characteristics:
|
|
|
|
* A NIC that exposes a card serial number via a PCIe VPD capability on its
|
|
physical or virtual function PCIe endpoints to both the hypervisor host
|
|
and the DPU host;
|
|
|
|
* Exposes the information about representor ports to applications running on
|
|
the SmartNIC DPU's CPU in a manner supported by one of the `OVN VIF Plug
|
|
Providers`_.
|
|
|
|
Nova configuration
|
|
------------------
|
|
|
|
Hypervisor hosts need to be configured to enable:
|
|
|
|
* `Nova PCI passthrough`_ for Nova Compute;
|
|
|
|
.. important::
|
|
|
|
For more information on other version requirements and limitations check
|
|
the `SR-IOV section of the Nova networking guide`_.
|
|
|
|
* SR-IOV virtual functions on selected physical functions provided by DPUs
|
|
to the hypervisor hosts.
|
|
|
|
In addition to the regular PCI device allow list configuration, the PCI device
|
|
specification must include the ``remote_managed`` tag as in the following
|
|
examples:
|
|
|
|
* Virtual networks without physical segments;
|
|
|
|
.. code-block:: ini
|
|
|
|
[pci]
|
|
passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}
|
|
|
|
* Physical networks (flat, VLAN) with a label:
|
|
|
|
.. code-block:: ini
|
|
|
|
[pci]
|
|
passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e", "physical_network": "dcfabric", "remote_managed": "true"}
|
|
|
|
.. note::
|
|
|
|
"dcfabric" is an arbitrary physnet name. In order for this to work it must
|
|
be specified consistenly in Nova config, during OVN configuraton when
|
|
specifying ``external_ids:ovn-bridge-mappings`` and during Neutron provider
|
|
network segment creation.
|
|
|
|
Auto-Discovery
|
|
--------------
|
|
|
|
When an instance with a ``remote-managed`` port is scheduled to a compute host
|
|
with a free remote-managed device, it claims it and supplies additional
|
|
information from that device about the NIC to Neutron so that it knows which
|
|
OVN chassis needs to handle an representor interface plugging and flow
|
|
programming. For PCI VFs this additional information includes:
|
|
|
|
* A card serial number from the NIC's VPD;
|
|
|
|
* PF mac address;
|
|
|
|
* VF logical number.
|
|
|
|
Neutron uses the card serial number to look up a chassis host name which is
|
|
needed for port binding to succeed and the rest is used by ``ovn-vif`` to set
|
|
up the matching representor port.
|
|
|
|
As a result, no direct communication or configuration is required between the
|
|
SmartNIC DPU host and the compute host in order to handle matching of compute
|
|
hosts to SmartNIC DPUs.
|
|
|
|
.. note::
|
|
|
|
Multiple DPUs per hypervisor host are possible to use, however, at the time
|
|
of writing, there is no way to indicate to Nova which VFs to choose via
|
|
Neutron port object attributes.
|
|
|
|
Having the OVN controller expose the SmartNIC DPU serial number is accomplished
|
|
by providing the serial number via the ``ovn-cms-options`` entry in
|
|
*external_ids* column of the SmartNIC DPU local *Open_vSwitch* table:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ ovs-vsctl set Open_vSwitch . external-ids:ovn-cms-options="card-serial-number=AB0123XX0042"
|
|
|
|
Launch an instance with remote managed port
|
|
-------------------------------------------
|
|
|
|
.. code-block:: bash
|
|
|
|
$ openstack port create \
|
|
--network network \
|
|
--vnic-type remote-managed \
|
|
port1
|
|
|
|
.. code-block:: bash
|
|
|
|
$ openstack server create \
|
|
--flavor 1 \
|
|
--nic port-id=port1
|
|
|
|
|
|
.. LINKS
|
|
.. _Off-path SmartNIC DPU Port Binding with OVN specification:
|
|
https://specs.openstack.org/openstack/neutron-specs/specs/yoga/off-path-smartnic-dpu-port-binding-with-ovn.html
|
|
.. _Nova Integration With Off-path Network Backends specification:
|
|
https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/integration-with-off-path-network-backends.html
|
|
.. _Nova PCI passthrough:
|
|
https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
|
|
.. _SR-IOV section of the Nova networking guide:
|
|
https://docs.openstack.org/nova/latest/admin/networking.html#sr-iov
|
|
.. _Open vSwitch: https://www.openvswitch.org/
|
|
.. _Open Virtual Network: https://www.ovn.org/
|
|
.. _OVN VIF: https://github.com/ovn-org/ovn-vif
|
|
.. _OVN VIF Plug Providers:
|
|
https://github.com/ovn-org/ovn-vif/tree/main/Documentation/topics/
|
|
vif-plug-providers
|