starlingx/kernel
Code Issues Proposed changes

libbnxt_re: Add support for rdma-core v37

Browse Source 
The version of rdma-core in StarlingX is based on version 37 provided by
Mellanox's OpenFabrics Enterprise Distribution, and the rdma-core
package's libibverbs has a library ABI version 34. Furthermore,
libbnxt_re's configure.ac script indicates that libbnxt_re is
(currently) compatible with rdma-core versions up to 35.

Despite the aforementioned, while preparing commit 9baff8aa88
("Introduce libbnxt_re version 220.0.5.0"), we were under the impression
that since the ABI version is 34, libbnxt_re would work fine, given that
it compiled without issues.

However, during run-time testing with the Linux-RDMA community's
perftest package, we observed that test programs such as "ib_send_bw"
would crash due to segmentation faults. Further debugging indicated that
libbnxt_re was calling an incorrect function by dereferencing a function
pointer in an "ops" structure, whose definition had been modified
between rdma-core v35 and v37 by inserting a new function pointer, which
caused some function pointer offsets to change. Furthermore, it was
noticed that libbnxt_re's build procedure was assuming that libbnxt_re
was being built against rdma-core v35, instead of what we had thought to
be v34. In conclusion, we had encountered an incompatibility.

Note that Broadcom has since released a newer driver and library bundle
(v221.1.28.0), but the newer version of libbnxt_re in that bundle does
not support rdma-core v37 either.

Due to what is discussed above, this commit patches libbnxt_re so that
it is compatible with rdma-core v37. We found that the API changes
between rdma-core v35 and v37 are Mellanox-specific and have no impact
on libbnxt_re. As a result, this commit only imports rdma-core v37.3's
headers and ensures that libbnxt_re can be built against the newer
headers.

Verification:
- An ISO image was built with this commit using an incremental and
  monolithic build procedure.
- The ISO image was installed and bootstrapped in low-latency All-in-One
  simplex mode on a server that has a quad-port Broadcom NetXtreme-E
  57504 network adapter.
- ib_send_bw and ib_send_lat utilities in the Linux-RDMA community's
  perftest package were observed to no longer crash due to segmentation
  faults with these changes.
- Basic tests were carried by connecting the third and fourth ports of
  the network adapter with a fiber and running traffic across the two
  ports. The test commands were as follows:

    # Server
    sudo ib_send_bw -F -D 10 -d bnxt_re2
    # Client
    sudo ib_send_bw -F -D 10 -d bnxt_re3 10.240.240.12

  According to the following commands, RDMA traffic was successfully
  processed by the aforementioned ports:

    ethtool -S enp101s0f2
    ethtool -S enp101s0f3
    sudo cat /sys/kernel/debug/bnxt_re/bnxt_re{2,3}/info

Story: 2009915
Task: 44916

Change-Id: I79528fb2d04e972dd69750f6bb3d132f9de5ada8
Fixes: 9baff8aa88 ("Introduce libbnxt_re version 220.0.5.0")
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
This commit is contained in:
M. Vefa Bicakci
2022-03-31 16:43:44 -04:00
parent db74d9176f
commit 4112936362
4 changed files with 8343 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
userspace/broadcom/libbnxt_re/centos/meta_patches/0002-libbnxt.spec-Add-support-for-rdma-core-v37.patch Normal file
View File

@@ -0,0 +1,38 @@
From a70be96c876117fb72c6cff9dc454fb22231c8b8 Mon Sep 17 00:00:00 2001
From: "M. Vefa Bicakci" <vefa.bicakci@windriver.com>
Date: Thu, 31 Mar 2022 16:37:32 -0400
Subject: [PATCH] libbnxt.spec: Add support for rdma-core v37
This modification to libbnxt.spec applies two patches to libbnxt_re, so
that it is compatible with rdma-core v37.
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
---
SPECS/libbnxt.spec | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/SPECS/libbnxt.spec b/SPECS/libbnxt.spec
index 992bda9e18ff..a509a80bdddf 100644
--- a/SPECS/libbnxt.spec
+++ b/SPECS/libbnxt.spec
@@ -18,6 +18,8 @@ BuildRequires: automake autoconf libtool
BuildRequires: kernel%{?bt_ext}-headers >= 5.10
Patch0001: 0001-Set-up-stdatomic.h-if-necessary.patch
+Patch0002: 0002-rc-compat-Add-headers-from-v37.3.patch
+Patch0003: 0003-configure.ac-Detect-rdma-core-v36-and-v37-too.patch
%description
libbnxt_re provides a device-specific userspace driver for Broadcom Netxtreme RoCE Adapters
@@ -35,6 +37,8 @@ application, which may be useful for debugging.
%prep
%setup -q -n %{name}-%{version}
%patch0001 -p1
+%patch0002 -p1
+%patch0003 -p1
%build
chmod a+x autogen.sh
--
2.29.2

1
userspace/broadcom/libbnxt_re/centos/meta_patches/PATCH_ORDER
View File

@@ -1 +1,2 @@
0001-libbnxt.spec-StarlingX-adaptations.patch
0002-libbnxt.spec-Add-support-for-rdma-core-v37.patch

8186
userspace/broadcom/libbnxt_re/centos/patches/0002-rc-compat-Add-headers-from-v37.3.patch Normal file
View File

File diff suppressed because it is too large Load Diff

118
userspace/broadcom/libbnxt_re/centos/patches/0003-configure.ac-Detect-rdma-core-v36-and-v37-too.patch Normal file
View File

@@ -0,0 +1,118 @@
From ffd1f85f70551c897e6e62b89acf8b170c1c82ea Mon Sep 17 00:00:00 2001
From: "M. Vefa Bicakci" <vefa.bicakci@windriver.com>
Date: Thu, 31 Mar 2022 16:14:22 -0400
Subject: [PATCH] configure.ac: Detect rdma-core v36 and v37 too
This commit makes the configure.ac script detect rdma-core v36 and v37
as well. Of these, we are only interested in v37, because Mellanox's
OpenFabrics Enterprise Distribution is based on rdma-core v37.
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
---
configure.ac | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 75 insertions(+), 1 deletion(-)
diff --git a/configure.ac b/configure.ac
index 9e4e980886d4..3ece437e664a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -68,6 +68,10 @@ m4_define([rcore_version],[m4_esyscmd(rc=$(grep -o \
_ibv_device_ops verbsincpath/include/infiniband/verbs.h | \
tail -1); \
if test ! -z "$rc"; then \
+ rcverv37=$(grep "IBV_WC_DRIVER3" \
+ verbsincpath/include/infiniband/verbs.h|tail -1); \
+ rcverv36=$(grep "ibv_query_qp_data_in_order" \
+ verbsincpath/include/infiniband/verbs.h|tail -1); \
rcverv35=$(grep "ibv_is_fork_initialized" \
verbsincpath/include/infiniband/verbs.h|tail -1); \
rcverv34=$(grep "ibv_reg_dmabuf_mr" \
@@ -90,7 +94,11 @@ m4_define([rcore_version],[m4_esyscmd(rc=$(grep -o \
verbsincpath/include/infiniband/verbs.h|tail -1); \
rcverv17=$(grep -o ibv_alloc_td \
verbsincpath/include/infiniband/verbs.h|tail -1); \
- if test ! -z "$rcverv35"; then \
+ if test ! -z "$rcverv37"; then \
+ echo -n "v37"; \
+ elif test ! -z "$rcverv36"; then \
+ echo -n "v36"; \
+ elif test ! -z "$rcverv35"; then \
echo -n "v35"; \
elif test ! -z "$rcverv34"; then \
echo -n "v34"; \
@@ -175,6 +183,72 @@ if test "$rc" != ""; then
AC_DEFINE([HAVE_IBV_CMD_MODIFY_QP_EX],[1],[ibv_cmd_modify_qp_ex supported])
RDMA_CORE_HEADERS=`echo -n __include_path__`
case "$(echo -n rcore_version)" in
+ v37)
+ AC_MSG_RESULT(yes using v37)
+ AC_DEFINE([IBV_WC_DRIVER3_IN_IBV_WC_OPCODE],[1],
+ [Rdma-Core Package has IBV_WC_DRIVER3 in ibv_wc_opcode enum])
+ AC_DEFINE([IBV_WC_DRIVER2_IN_IBV_WC_OPCODE],[1],
+ [Rdma-Core Package has IBV_WC_DRIVER2 in ibv_wc_opcode enum])
+ AC_DEFINE([IBV_QUERY_QP_DATA_IN_ORDER_IN_CONTEXT_OPS],[1],
+ [Rdma-Core Package has query_qp_data_in_order in verbs_context_ops])
+ AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
+ [Rdma-Core Package has only query_device_ex])
+ AC_DEFINE([IBV_CMD_MODIFY_QP_EX_HAS_7_ARG],[1],
+ [Rdma-Core Package ibv_cmd_modify_qp_ex has 7 arguments])
+ AC_DEFINE([IBV_FREE_CONTEXT_IN_CONTEXT_OPS],[1],
+ [Rdma-Core Package has free context in ibv_context_ops])
+ AC_DEFINE([REG_MR_VERB_HAS_5_ARG],[1],
+ [Rdma-Core Package reg_mr has 5 argument])
+ AC_DEFINE([IBV_CMD_ALLOC_MW_HAS_1_ARG],[1],
+ [Rdma-Core Package ibv_cmd_alloc_mw has single argument])
+ AC_DEFINE([PROVIDER_DRIVER_HAS_2_ARGS],[1],
+ [Rdma-Core Package PROVIDER_DRIVER macro has 2 arguments])
+ AC_DEFINE([ALLOC_CONTEXT_HAS_PRIVATE_DATA],[1],
+ [Rdma-Core Package alloc_context has private data])
+ AC_DEFINE([VERBS_MR_DEFINED],[1],
+ [Rdma-Core Package has verbs_mr defined])
+ AC_DEFINE([VERBS_INIT_AND_ALLOC_CONTEXT_HAS_5_ARG],[1],
+ [Rdma-Core Package has 5 Arg for verbs_init_and_alloc_context macro])
+ AC_DEFINE([RCP_HAS_PROVIDER_DRIVER],[1],
+ [Rdma-Core Package has PROVIDER_DRIVER macro])
+ AC_DEFINE([RCP_USE_IB_UVERBS],[1],
+ [Rdma-Core Package uses IB user verbs API])
+ AC_DEFINE([RCP_USE_ALLOC_CONTEXT],[1],
+ [Rdma-Core Package uses alloc_context instead of init_context])
+ rc=`grep -o ibv_read_sysfs_file src/rc-compat/v37/driver.h|tail -1`
+ ;;
+ v36)
+ AC_MSG_RESULT(yes using v36)
+ AC_DEFINE([IBV_WC_DRIVER2_IN_IBV_WC_OPCODE],[1],
+ [Rdma-Core Package has IBV_WC_DRIVER2 in ibv_wc_opcode enum])
+ AC_DEFINE([IBV_QUERY_QP_DATA_IN_ORDER_IN_CONTEXT_OPS],[1],
+ [Rdma-Core Package has query_qp_data_in_order in verbs_context_ops])
+ AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
+ [Rdma-Core Package has only query_device_ex])
+ AC_DEFINE([IBV_CMD_MODIFY_QP_EX_HAS_7_ARG],[1],
+ [Rdma-Core Package ibv_cmd_modify_qp_ex has 7 arguments])
+ AC_DEFINE([IBV_FREE_CONTEXT_IN_CONTEXT_OPS],[1],
+ [Rdma-Core Package has free context in ibv_context_ops])
+ AC_DEFINE([REG_MR_VERB_HAS_5_ARG],[1],
+ [Rdma-Core Package reg_mr has 5 argument])
+ AC_DEFINE([IBV_CMD_ALLOC_MW_HAS_1_ARG],[1],
+ [Rdma-Core Package ibv_cmd_alloc_mw has single argument])
+ AC_DEFINE([PROVIDER_DRIVER_HAS_2_ARGS],[1],
+ [Rdma-Core Package PROVIDER_DRIVER macro has 2 arguments])
+ AC_DEFINE([ALLOC_CONTEXT_HAS_PRIVATE_DATA],[1],
+ [Rdma-Core Package alloc_context has private data])
+ AC_DEFINE([VERBS_MR_DEFINED],[1],
+ [Rdma-Core Package has verbs_mr defined])
+ AC_DEFINE([VERBS_INIT_AND_ALLOC_CONTEXT_HAS_5_ARG],[1],
+ [Rdma-Core Package has 5 Arg for verbs_init_and_alloc_context macro])
+ AC_DEFINE([RCP_HAS_PROVIDER_DRIVER],[1],
+ [Rdma-Core Package has PROVIDER_DRIVER macro])
+ AC_DEFINE([RCP_USE_IB_UVERBS],[1],
+ [Rdma-Core Package uses IB user verbs API])
+ AC_DEFINE([RCP_USE_ALLOC_CONTEXT],[1],
+ [Rdma-Core Package uses alloc_context instead of init_context])
+ rc=`grep -o ibv_read_sysfs_file src/rc-compat/v36/driver.h|tail -1`
+ ;;
v35)
AC_MSG_RESULT(yes using v35)
AC_DEFINE([VERBS_ONLY_QUERY_DEVICE_EX_DEFINED],[1],
--
2.29.2