From 0ae9f6bd537b726950712ac1ee0d5a6a4f6f82f5 Mon Sep 17 00:00:00 2001 From: "M. Vefa Bicakci" Date: Wed, 16 Feb 2022 12:07:09 -0500 Subject: [PATCH] Add Broadcom NetXtreme-E drivers This commit introduces Broadcom's bnxt_en and bnxt_re out-of-tree drivers, version 220.0.13.0, to StarlingX, and disables the in-tree bnxt_en driver. (For clarity, the main driver package file name is "bcm_220.0.83.0.tar.gz".) There is a need to build Broadcom's driver against the version of the Infiniband/RDMA code base bundled with Mellanox OFED (OpenFabrics Enterprise Distribution), because the in-tree kernel Infiniband/RDMA modules are disabled, and Mellanox OFED's out-of-tree Infiniband/RDMA modules replace the in-tree versions. A custom spec file was used, despite the fact that the Broadcom device driver package includes a source RPM file. This was necessary due to two reasons: - The source RPM file has whitespace between its field labels (such as Name, Version, Release, and so on) and the values of these fields. (Example: "Name : bnxt_en") This unfortunately breaks "build-pkgs --edit", which cannot handle the whitespace during the extraction of the source RPM. - More importantly, the spec file included by Broadcom has the following lines in its %prep section, which does not use the %setup macro: %prep #%setup mkdir -p source tar -xzvf %{SOURCE0} -C source/ mv -f source/netxtreme-%name-%{version}-220.0.13.0/* source/ rm -rf source/netxtreme-%name-%{version}-220.0.13.0/ cp ./source/%{bnxt_re_driver}/README.TXT ./source/%{bnxt_re_driver}/%{bnxt_re_driver}_README.TXT cp ./source/%{bnxt_en_driver}/README.TXT ./source/%{bnxt_en_driver}/%{bnxt_en_driver}_README.TXT mkdir -p obj This is problematic, because the "build-pkgs --edit" procedure extracts the source RPM multiple times into the same directory to ensure that the extraction is stable. However, the %prep section quoted above does not remove the 'source' directory before attempting to populate it. (Unfortunately, this issue is not patchable using a meta-patch, because the extraction occurs even before the application of meta-patches.) This in turn triggers the following errors: + mv -f source/netxtreme-bnxt_en-1.10.2-220.0.13.0/Makefile \ source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_en \ source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_en-1.10.2-220.0.13.0 \ source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_re \ source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_re-220.0.12.0 \ source/ mv: cannot move 'source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_en-1.10.2-220.0.13.0' \ to 'source/bnxt_en-1.10.2-220.0.13.0': Directory not empty mv: cannot move 'source/netxtreme-bnxt_en-1.10.2-220.0.13.0/bnxt_re-220.0.12.0' \ to 'source/bnxt_re-220.0.12.0': Directory not empty error: Bad exit status from .../std/srpm_work/bnxt_en/var/tmp/rpm-tmp.cDTpOP (%prep) Bad exit status from .../std/srpm_work/bnxt_en/var/tmp/rpm-tmp.cDTpOP (%prep) The changes made to the spec file are as follows: - Reliance on Red Hat's kernel module packaging RPM spec macros (such as %kernel_module_package) is removed, because the aforementioned macros make use of the kmodtool helper program, which automatically generates an RPM post-installation and post-uninstallation sections making use of the weak-modules utility, which StarlingX does not use/support. It is not possible to disable this behaviour of the kmodtool script ("/usr/lib/rpm/redhat/kmodtool") with a command line option or an environment variable. - The spec file is reorganized to build the kernel module against only a single kernel "flavour", as StarlingX does not use the kernel flavour concept. - The %build process is modified so that instead of using the top-level makefile (which does not take certain variables as arguments), the bnxt_en- and bnxt_re-specific makefiles are called with make separately. This allows passing KDIR, KVER and OFED_VERSION make variables as command line arguments to make. - The use of the devtoolset-8 software collection was added to align to the kernel's build toolchain. - The spec file is modified to detect the OFED version and pass the version to the bnxt_re makefile as the OFED_VERSION environment variable. This is necessary to automatically adapt to the version of OFED installed in the build environment. - The %post and %postun scriptlets are modified to make them similar to the ones used by i40e, which passes the target kernel's version to depmod. The original %post only had 'depmod -a', which can be problematic when installing the package for a kernel that is not currently running. - Kernel module signing was added to %install. This shell snippet was adapted from i40e. - Finally, retpolines were enabled by default, given that StarlingX enables retpolines in its kernel packages as well. In terms of source code modifications, the following changes were made to bnxt_en and bnxt_re: - The bnxt_en and bnxt_re kernel modules are patched to make use of the irq_update_affinity_hint function instead of the irq_set_affinity_hint function, so that the irqaffinity= kernel command line option takes effect for the affinities of the IRQs managed by the two drivers. However, it should be noted the changes in bnxt_en are not truly necessary, because pre-existing conditional compilation directives leave out the modified code when targeting kernels >= v4.15 or kernels that have the "IRQ matrix allocator" feature. The changes are kept for completeness, nevertheless. For further details, please see the patch description. - A compilation failure caused by the definition of a placeholder macro aimed to stand in for the bpf_warn_invalid_xdp_action function in the kernel tree is resolved by removing the macro definition. Testing: - An ISO image was successfully built with a monolithic incremental build procedure. - Using a virtual machine in All-in-One simplex configuration, the bnxt_en and bnxt_re kernel modules were confirmed to be successfully loaded manually using, for example, "modprobe -v -a bnxt_en bnxt_re". Runtime tests for these drivers were unfortunately very limited, because the necessary hardware has not yet been acquired. We intend to resolve integration issues when hardware is acquired. Story: 2009915 Task: 44762 Depends-On: https://review.opendev.org/c/starlingx/kernel/+/833463 Depends-On: https://review.opendev.org/c/starlingx/tools/+/833462 Change-Id: Ieb4147335410160a781382b0b6b01e2015468a3b Signed-off-by: M. Vefa Bicakci --- centos_extra_downloads.lst | 1 + centos_iso_image.inc | 4 + centos_pkg_dirs | 1 + centos_pkg_dirs_rt | 1 + centos_tarball-dl.lst | 1 + kernel-modules/bnxt_en/centos/bnxt_en.spec | 135 ++++++++++++++++++ kernel-modules/bnxt_en/centos/build_srpm.data | 11 ++ ...-Adapt-to-mlnx-ofa_kernel-for-Starli.patch | 117 +++++++++++++++ ...bnxt_compat.h-Fix-up-a-build-failure.patch | 45 ++++++ ...bnxt_re-Use-irq_update_affinity_hint.patch | 134 +++++++++++++++++ .../files/kernel-x86_64-rt.config.tis_extra | 2 + .../files/kernel-x86_64.config.tis_extra | 2 + 12 files changed, 454 insertions(+) create mode 100644 kernel-modules/bnxt_en/centos/bnxt_en.spec create mode 100644 kernel-modules/bnxt_en/centos/build_srpm.data create mode 100644 kernel-modules/bnxt_en/centos/patches/0001-bnxt_re-Makefile-Adapt-to-mlnx-ofa_kernel-for-Starli.patch create mode 100644 kernel-modules/bnxt_en/centos/patches/0002-bnxt_en-bnxt_compat.h-Fix-up-a-build-failure.patch create mode 100644 kernel-modules/bnxt_en/centos/patches/0003-bnxt_en-bnxt_re-Use-irq_update_affinity_hint.patch diff --git a/centos_extra_downloads.lst b/centos_extra_downloads.lst index 1e161858..65024730 100644 --- a/centos_extra_downloads.lst +++ b/centos_extra_downloads.lst @@ -4,3 +4,4 @@ mstflint-4.16.0-1.55103.src.rpm rdma-core-55mlnx37-1.55103.src.rpm opae-intel-fpga-driver-2.0.1-10.src.rpm ice_comms-1.3.31.0.zip +netxtreme-bnxt_en-1.10.2-220.0.13.0.tar.gz diff --git a/centos_iso_image.inc b/centos_iso_image.inc index c355ce2c..8f4f92ae 100644 --- a/centos_iso_image.inc +++ b/centos_iso_image.inc @@ -60,3 +60,7 @@ rdma-core # mellanox firmware tool mstflint + +# Broadcom device driver +kmod-bnxt_en +kmod-bnxt_en-rt diff --git a/centos_pkg_dirs b/centos_pkg_dirs index f43e9505..db6c85cc 100644 --- a/centos_pkg_dirs +++ b/centos_pkg_dirs @@ -1,4 +1,5 @@ kernel-std +kernel-modules/bnxt_en kernel-modules/intel-iavf kernel-modules/intel-i40e kernel-modules/intel-ice diff --git a/centos_pkg_dirs_rt b/centos_pkg_dirs_rt index e7765b06..b83d93db 100644 --- a/centos_pkg_dirs_rt +++ b/centos_pkg_dirs_rt @@ -1,4 +1,5 @@ kernel-rt +kernel-modules/bnxt_en kernel-modules/mlnx-ofa_kernel kernel-modules/intel-iavf kernel-modules/intel-i40e diff --git a/centos_tarball-dl.lst b/centos_tarball-dl.lst index b2bbbbda..4b4760ab 100644 --- a/centos_tarball-dl.lst +++ b/centos_tarball-dl.lst @@ -14,3 +14,4 @@ opae-intel-fpga-driver-2.0.1-8.tar.gz#opae-intel-fpga-driver-2.0.1-8#https://git !dpdk-kmods-2a9f0f72a2d926382634cf8f1de10e1acf57542b.tar.gz#dpdk-kmods#git://dpdk.org/dpdk-kmods/#git#2a9f0f72a2d926382634cf8f1de10e1acf57542b## !OPAE_1.3.7-5_el7.zip#OPAE#https://github.com/OPAE/opae-sdk/releases/download/1.3.7-5/OPAE_1.3.7-5_el7.zip#http_script##unused !800-Series-DDP-Comms-Package-1.3.31.0.zip#unused-field#https://downloadmirror.intel.com/713853/800%20Series%20DDP%20Comms%20Package%201.3.31.0.zip#http_script##unused +!bcm_220.0.83.0.tar.gz#bcm_220.0.83.0#https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_220.0.83.0/bcm_220.0.83.0.tar.gz#http_script#85bdfc30b4bd3e184e3b57a48055c11085e3b97593f7b4a8347fa50a9d571336#unused diff --git a/kernel-modules/bnxt_en/centos/bnxt_en.spec b/kernel-modules/bnxt_en/centos/bnxt_en.spec new file mode 100644 index 00000000..258df20e --- /dev/null +++ b/kernel-modules/bnxt_en/centos/bnxt_en.spec @@ -0,0 +1,135 @@ +%if "%{?_tis_build_type}" == "rt" +%define bt_ext -rt +%else +%undefine bt_ext +%endif + +%define kmod_name bnxt_en +%define upstream_release 220.0.13.0 + +## Condition to check if retpolines are needed. +## Default: retpolines are enabled. +%bcond_without retpoline + +%define kernel_module_package_buildreqs kernel%{?bt_ext}-devel + +# Disable the building of debug package(s). +%define debug_package %{nil} + +Name: kmod-%{kmod_name}%{?bt_ext} +Version: 1.10.2 +Release: %{upstream_release}%{?_tis_dist}.%{tis_patch_ver} +Group: System Environment/Base +Packager: StarlingX Community +Vendor: Broadcom Inc +URL: https://www.broadcom.com/support/download-search?pg=&pf=&pn=&pa=&po=&dk=bnxt_en&pl= +License: GPLv2 +Summary: Broadcom NetXtreme Gigabit Ethernet Driver + +Provides: %{kmod_name}-kmod = %{?epoch:%{epoch}:}%{version}-%{release} +BuildRequires: devtoolset-8-build +BuildRequires: devtoolset-8-binutils +BuildRequires: devtoolset-8-gcc +BuildRequires: devtoolset-8-make +BuildRequires: kernel%{?bt_ext}-devel findutils +BuildRequires: mlnx-ofa_kernel%{?bt_ext}-devel mlnx-ofa_kernel%{?bt_ext}-source +Requires: mlnx-ofa_kernel%{?bt_ext}-modules +Requires(post): /usr/sbin/depmod +Requires(postun): /usr/sbin/depmod + +Source0: netxtreme-%{kmod_name}-%{version}-%{upstream_release}.tar.gz + +Patch0001: 0001-bnxt_re-Makefile-Adapt-to-mlnx-ofa_kernel-for-Starli.patch +Patch0002: 0002-bnxt_en-bnxt_compat.h-Fix-up-a-build-failure.patch +Patch0003: 0003-bnxt_en-bnxt_re-Use-irq_update_affinity_hint.patch + +#define some build variables required +%define bnxt_en_driver bnxt_en +%define bnxt_re_driver bnxt_re + +%define kversion %(rpm -q kernel%{?bt_ext}-devel | sort --version-sort | tail -1 | sed 's/kernel%{?bt_ext}-devel-//') + +%description +This package provides the %{bnxt_en_driver} and %{bnxt_re_driver} kernel +modules built for the Linux kernel for the %{_target_cpu} family of processors. + +%prep +%autosetup -p 1 -n netxtreme-%{kmod_name}-%{version}-%{upstream_release} +cp %{bnxt_re_driver}/README.TXT %{bnxt_re_driver}/%{bnxt_re_driver}_README.TXT +cp %{bnxt_en_driver}/README.TXT %{bnxt_en_driver}/%{bnxt_en_driver}_README.TXT + +%build +%if 0%{?rhel} == 7 +source scl_source enable devtoolset-8 || : +%endif + +export EXTRA_CFLAGS+='-DVERSION=\"%{version}\"' + +%if %{with retpoline} + find . -name *.c -print0 | xargs -0 sed -i '/MODULE_LICENSE(/a MODULE_INFO(retpoline, "Y");' +%endif + +# Regenerate MANIFEST HASH +if [ -e bnxt_en/MANIFEST ]; then + pushd bnxt_en >/dev/null + sha512sum $(cat MANIFEST | cut -c 131-) > MANIFEST + popd >/dev/null +fi + +%{__make} %{_smp_mflags} -C bnxt_en \ + KVER=%{kversion} \ + KDIR=%{_usrsrc}/kernels/%{kversion} + +OFED_VERSION="$(ls -1vd %{_usrsrc}/ofa_kernel-* | xargs -r -n1 basename | \ + sed -e 's@^ofa_kernel-@@' | sort --version-sort | tail -n1)" +if test -z "${OFED_VERSION}"; then + echo "Error: Could not detect the OFED version" + exit 1 +fi + +%{__make} %{_smp_mflags} -C bnxt_re \ + KVER=%{kversion} \ + KDIR=%{_usrsrc}/kernels/%{kversion} \ + OFED_VERSION="${OFED_VERSION}" + +%install +export KVER=%{kversion} + +dest_dir_bnxt_en=${RPM_BUILD_ROOT}/lib/modules/${KVER}/extra/%{kmod_name}/ +src_dir_bnxt_en=$(find . -name %{bnxt_en_driver}.ko) +dest_dir_bnxt_re=${RPM_BUILD_ROOT}/lib/modules/${KVER}/extra/%{kmod_name}/drivers/infiniband/hw/%{bnxt_re_driver}/ +src_dir_bnxt_re=$(find . -name %{bnxt_re_driver}.ko) +mkdir -p ${dest_dir_bnxt_en} +mkdir -p ${dest_dir_bnxt_re} +install -m 744 ${src_dir_bnxt_en} ${dest_dir_bnxt_en} +install -m 744 ${src_dir_bnxt_re} ${dest_dir_bnxt_re} + +### Adapted from i40e +# Strip the modules(s). +find %{buildroot} -type f -name \*.ko -exec %{__strip} --strip-debug \{\} \; + +# Always Sign the modules(s). +# If the module signing keys are not defined, define them here. +%{!?privkey: %define privkey /usr/src/kernels/%{kversion}/signing_key.pem} +%{!?pubkey: %define pubkey /usr/src/kernels/%{kversion}/signing_key.x509} +for module in $(find %{buildroot} -type f -name \*.ko); +do /usr/src/kernels/%{kversion}/scripts/sign-file \ + sha256 %{privkey} %{pubkey} $module; +done + +%files +%defattr(644,root,root,755) +/lib/modules/%{kversion}/extra/%{kmod_name}/ + +%post +if [ -e "/boot/System.map-%{kversion}" ]; then + /usr/sbin/depmod -aeF "/boot/System.map-%{kversion}" "%{kversion}" > /dev/null || : +fi + +%postun +/usr/sbin/depmod -a >/dev/null 2>&1 || true + +%clean +rm -rf %{buildroot} + +%changelog diff --git a/kernel-modules/bnxt_en/centos/build_srpm.data b/kernel-modules/bnxt_en/centos/build_srpm.data new file mode 100644 index 00000000..1409dbf9 --- /dev/null +++ b/kernel-modules/bnxt_en/centos/build_srpm.data @@ -0,0 +1,11 @@ +COPY_LIST=" \ + $PATCHES_BASE/* \ + $STX_BASE/downloads/netxtreme-bnxt_en-1.10.2-220.0.13.0.tar.gz \ + " + +OPT_DEP_LIST_FOR_BUILD_TYPE[std]=$GIT_BASE/kernel-std +OPT_DEP_LIST_FOR_BUILD_TYPE[rt]=$GIT_BASE/kernel-rt +BASE_SRCREV_FOR_PATH[$GIT_BASE/kernel-std]=1378583af6e040bc974f67850cd44e0c62d42f81 +BASE_SRCREV_FOR_PATH[$GIT_BASE/kernel-rt]=1378583af6e040bc974f67850cd44e0c62d42f81 +PKG_BASE_SRCREV=1378583af6e040bc974f67850cd44e0c62d42f81 +TIS_PATCH_VER=PKG_GITREVCOUNT+OTHER_GITREVCOUNT diff --git a/kernel-modules/bnxt_en/centos/patches/0001-bnxt_re-Makefile-Adapt-to-mlnx-ofa_kernel-for-Starli.patch b/kernel-modules/bnxt_en/centos/patches/0001-bnxt_re-Makefile-Adapt-to-mlnx-ofa_kernel-for-Starli.patch new file mode 100644 index 00000000..4843ed30 --- /dev/null +++ b/kernel-modules/bnxt_en/centos/patches/0001-bnxt_re-Makefile-Adapt-to-mlnx-ofa_kernel-for-Starli.patch @@ -0,0 +1,117 @@ +From b4de4f0d87331a3df820ca78a7196b41a5ee74d5 Mon Sep 17 00:00:00 2001 +From: "M. Vefa Bicakci" +Date: Fri, 4 Feb 2022 18:27:53 -0500 +Subject: [PATCH] bnxt_re/Makefile: Adapt to mlnx-ofa_kernel for StarlingX + +This patch adapts the bnxt_re kernel module's Makefile to +mlnx-ofa_kernel for StarlingX. + +Here is a summary of the changes: + +- $(shell uname -r) is replaced with $(KVER) to ensure that the target + kernel version is selected instead of the build host's kernel version. + +- To target Mellanox's OFED distribution instead of compat-rdma, the + references to "/usr/src/compat-rdma" are replaced with + "/usr/src/ofa_kernel". + +- Mellanox's OFED distribution has version numbers in the form 5.x, so + the Makefile version match patterns are adjusted. + +- "AUTOCONF_H += -include $(LINUXSRC)/include/linux/kconfig.h" line is + added avoid the following kernel module build errors indicating that + IS_ENABLED is unintendedly undefined: + + include/asm-generic/bitops/instrumented-non-atomic.h: In function \ + '__instrument_read_write_bitop': + include/asm-generic/bitops/instrumented-non-atomic.h:63:6: error: \ + implicit declaration of function 'IS_ENABLED' \ + [-Werror=implicit-function-declaration] + if (IS_ENABLED(CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC)) { + ^~~~~~~~~~ + +- The line that reads + "KBUILD_EXTRA_SYMBOLS := $(BNXT_PEER_MEM_INC)/Module.symvers" + is guarded to avoid the following build warning, which is encountered + as BNXT_PEER_MEM_INC is not defined by default: + WARNING: Symbol version dump "/Module.symvers" is missing. + +- OFA_BUILD_PATH and OFA_KERNEL_PATH are adjusted to refer to the kernel + source code belonging to the target kernel version in + "/usr/src/kernels/", because StarlingX's build environment does not + populate /lib/modules// with symbolic links for kernel source + and build trees. + +Signed-off-by: M. Vefa Bicakci +--- + bnxt_re-220.0.12.0/Makefile | 30 ++++++++++++++++++------------ + 1 file changed, 18 insertions(+), 12 deletions(-) + +diff --git a/bnxt_re-220.0.12.0/Makefile b/bnxt_re-220.0.12.0/Makefile +index 532fb8025465..d13a61379acb 100644 +--- a/bnxt_re-220.0.12.0/Makefile ++++ b/bnxt_re-220.0.12.0/Makefile +@@ -32,7 +32,7 @@ ifeq ($(shell ls /lib/modules/$(KVER)/build > /dev/null 2>&1 && echo build),) + # SuSE source RPMs + _KVER=$(shell echo $(KVER) | cut -d "-" -f1,2) + _KFLA=$(shell echo $(KVER) | cut -d "-" -f3) +- _ARCH=$(shell file -b /lib/modules/$(shell uname -r)/build | cut -d "/" -f5) ++ _ARCH=$(shell file -b /lib/modules/$(KVER)/build | cut -d "/" -f5) + ifeq ($(_ARCH),) + _ARCH=$(__ARCH) + endif +@@ -90,34 +90,40 @@ ifeq ($(OFED_VERSION), ) + endif + + #find OFED version and compat-includes +-ofed_major=$(filter OFED-3.% OFED-4.%, $(OFED_VERSION)) ++ofed_major=$(filter OFED-3.% OFED-4.% 5.%, $(OFED_VERSION)) + ifneq ($(ofed_major), ) +-exists=$(shell if [ -e /usr/src/compat-rdma$(OFED_VERSION) ];\ ++exists=$(shell if [ -e /usr/src/ofa_kernel-$(OFED_VERSION) ];\ + then echo y; fi) + ifeq ($(exists), ) +-$(shell ln -s /usr/src/compat-rdma\ +- /usr/src/compat-rdma$(OFED_VERSION)) ++$(shell ln -s /usr/src/ofa_kernel\ ++ /usr/src/ofa_kernel-$(OFED_VERSION)) + endif +-OFA_BUILD_PATH=/usr/src/compat-rdma$(OFED_VERSION) +-OFA_KERNEL_PATH=/usr/src/compat-rdma$(OFED_VERSION) ++OFA_BUILD_PATH=/usr/src/ofa_kernel/default ++OFA_KERNEL_PATH=/usr/src/ofa_kernel-$(OFED_VERSION)/source + EXTRA_CFLAGS += -DOFED_3_x +-ofed_4_17_x=$(filter OFED-4.17%, $(ofed_major)) ++ofed_4_17_x=$(filter OFED-4.17% 5.%, $(ofed_major)) + ifneq ($(ofed_4_17_x), ) + EXTRA_CFLAGS += -D__OFED_BUILD__ + endif + EXTRA_CFLAGS += -include $(OFA_KERNEL_PATH)/include/linux/compat-2.6.h + + AUTOCONF_H = -include $(shell /bin/ls -1 $(LINUX)/include/*/autoconf.h 2> /dev/null | head -1) ++AUTOCONF_H += -include $(LINUXSRC)/include/linux/kconfig.h + endif #end non 3.x OFED ++ ++ifneq ($(BNXT_PEER_MEM_INC),) + KBUILD_EXTRA_SYMBOLS := $(BNXT_PEER_MEM_INC)/Module.symvers ++else ++KBUILD_EXTRA_SYMBOLS := ++endif + + ifeq (OFED-NATIVE, $(findstring OFED-NATIVE, $(OFED_VERSION))) + ifeq ($(KERNELRELEASE),) +-OFA_BUILD_PATH =/lib/modules/`uname -r`/build/ +-OFA_KERNEL_PATH=/lib/modules/`uname -r`/source/ ++OFA_BUILD_PATH=/usr/src/kernels/$(KVER)/ ++OFA_KERNEL_PATH=/usr/src/kernels/$(KVER)/ + else +-OFA_BUILD_PATH =/lib/modules/$(KERNELRELEASE)/build/ +-OFA_KERNEL_PATH=/lib/modules/$(KERNELRELEASE)/source/ ++OFA_BUILD_PATH=/usr/src/kernels/$(KERNELRELEASE)/ ++OFA_KERNEL_PATH=/usr/src/kernels/$(KERNELRELEASE)/ + endif + else + # Add OFED symbols only if external OFED is used +-- +2.29.2 + diff --git a/kernel-modules/bnxt_en/centos/patches/0002-bnxt_en-bnxt_compat.h-Fix-up-a-build-failure.patch b/kernel-modules/bnxt_en/centos/patches/0002-bnxt_en-bnxt_compat.h-Fix-up-a-build-failure.patch new file mode 100644 index 00000000..6f3fe3fe --- /dev/null +++ b/kernel-modules/bnxt_en/centos/patches/0002-bnxt_en-bnxt_compat.h-Fix-up-a-build-failure.patch @@ -0,0 +1,45 @@ +From c6e38e23c1a2d45efcab73793ebe28f8fcb504e4 Mon Sep 17 00:00:00 2001 +From: "M. Vefa Bicakci" +Date: Fri, 4 Feb 2022 19:44:06 -0500 +Subject: [PATCH] bnxt_en/bnxt_compat.h: Fix up a build failure + +This commit resolves the following build failure: + + In file included from ./include/net/sock.h:59, + from ./include/linux/tcp.h:19, + from ./include/linux/ipv6.h:87, + from ./include/net/dsfield.h:12, + from ./include/net/ip_tunnels.h:13, + from ./include/net/dst_metadata.h:6, + from /builddir/build/BUILD/netxtreme-bnxt_en-1.10.2-219.0.55.0/bnxt_en-1.10.2-219.0.55.0/bnxt.h:30, + from /builddir/build/BUILD/netxtreme-bnxt_en-1.10.2-219.0.55.0/bnxt_en-1.10.2-219.0.55.0/bnxt_hwrm.c:26: + ./include/linux/filter.h:980:1: error: useless type name in empty declaration [-Werror] + void bpf_warn_invalid_xdp_action(u32 act); + ^~~~ + cc1: all warnings being treated as errors + +The problem arises from the fact that the function +bpf_warn_invalid_xdp_action is already defined in the kernel. Having a +placeholder macro definition with the same name causes the compilation +warning (which turns into an error due to -Werror). + +Signed-off-by: M. Vefa Bicakci +--- + bnxt_en-1.10.2-220.0.13.0/bnxt_compat.h | 1 - + 1 file changed, 1 deletion(-) + +diff --git a/bnxt_en-1.10.2-220.0.13.0/bnxt_compat.h b/bnxt_en-1.10.2-220.0.13.0/bnxt_compat.h +index 63d27f2e8979..aeb15fc9ab2a 100644 +--- a/bnxt_en-1.10.2-220.0.13.0/bnxt_compat.h ++++ b/bnxt_en-1.10.2-220.0.13.0/bnxt_compat.h +@@ -1596,7 +1596,6 @@ enum xdp_action { + + #ifndef HAVE_BPF_TRACE + #define trace_xdp_exception(dev, xdp_prog, act) +-#define bpf_warn_invalid_xdp_action(act) + #endif + + #ifdef HAVE_XDP_RXQ_INFO +-- +2.29.2 + diff --git a/kernel-modules/bnxt_en/centos/patches/0003-bnxt_en-bnxt_re-Use-irq_update_affinity_hint.patch b/kernel-modules/bnxt_en/centos/patches/0003-bnxt_en-bnxt_re-Use-irq_update_affinity_hint.patch new file mode 100644 index 00000000..16107d03 --- /dev/null +++ b/kernel-modules/bnxt_en/centos/patches/0003-bnxt_en-bnxt_re-Use-irq_update_affinity_hint.patch @@ -0,0 +1,134 @@ +From f278ab69adb9e59041d2b8e78b6005587f94d0e3 Mon Sep 17 00:00:00 2001 +From: "M. Vefa Bicakci" +Date: Wed, 2 Mar 2022 14:14:12 -0500 +Subject: [PATCH] bnxt_en, bnxt_re: Use irq_update_affinity_hint + +This commit updates the bnxt_en and bnxt_re device drivers to use the +irq_update_affinity_hint function instead of the irq_set_affinity_hint +function, so that StarlingX's CPU affinity requirements for interrupts +are met. (Note that the StarlingX kernel has been patched to add the +irq_update_affinity_hint API function, which originally did not exist in +kernel version 5.10.) + +irq_set_affinity_hint sets the IRQ affinity hint as well as the actual +IRQ affinity, which causes the IRQ affinity to not align with the value +of the irqaffinity= command line option. + +StarlingX's user-space initialization scripts set the IRQ affinities +too, but if a network interface is initialized after the completion of +the initialization scripts, then the network adapter's preferred IRQ +affinity passed to irq_set_affinity_hint takes effect. The latter is +undesirable if the CPU affinity of an interrupt includes CPUs that are +in the set of isolated CPUs configured by StarlingX, as isolated CPUs +should not service unrelated interrupts. + +To resolve the aforementioned issues, this patch updates the bnxt_en and +bnxt_re drivers to use irq_update_affinity_hint, which only sets the IRQ +affinity hint, and not the actual IRQ affinity. This allows interrupts +to be affined to the CPUs listed in the irqaffinity= command line +argument by default, and eliminates the need to set the IRQ affinities +after the initialization of network interfaces. + +Finally, please note that the changes in bnxt_en were confirmed to not +be compiled with StarlingX's 5.10-based kernel, because of the following +conditional compilation pre-processor directives that are above the +irq_set_affinity_hint/irq_update_affinity_hint call site: + + #if (!defined(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) || \ + KERNEL_VERSION(4, 15, 0) > LINUX_VERSION_CODE) && \ + (defined(HAVE_CPUMASK_LOCAL_FIRST) || \ + defined(HAVE_CPUMASK_LOCAL_SPREAD)) + + /* ... */ + cpumask_set_cpu(cpumask_local_spread(i, numa_node), + irq->cpu_mask); + /* ... */ + rc = irq_update_affinity_hint(irq->vector, irq->cpu_mask); + /* ... */ + +The IRQ matrix allocator is enabled in StarlingX's kernel, which also +has a version greater than 4.15. 4.15 happens to be the kernel version +with which the IRQ matrix allocator was introduced. In other words, the +call site in question is not compiled at all with the out-of-tree +bnxt_en driver. The in-tree version of bnxt_en does not have conditional +compilation directives around the same code (i.e., the code is always +compiled in). + +Our imperfect understanding of the intent is that the out-of-tree +bnxt_en driver's authors decided to rely on the IRQ matrix allocator's +default IRQ vector distribution behaviour (which appears to occur at a +lower level) instead of setting the IRQ affinity hints explicitly. + +Thankfully, the lack of a call irq_update_affinity_hint does not prevent +the irqaffinity= command line argument from taking effect for such +interrupts. This was confirmed with a virtual machine and the e1000e +driver, which also manages multiple MSI-X interrupts and which also does +not explicitly set the IRQ affinities nor affinity hints. In our +experiments, the IRQ affinities of the emulated Intel network adapter +correctly matched the CPU list set by the irqaffinity= command line +argument. + +Despite what is discussed above, the code changes in bnxt_en are kept +for completeness. + +Signed-off-by: M. Vefa Bicakci +--- + bnxt_en-1.10.2-220.0.13.0/bnxt.c | 6 +++--- + bnxt_re-220.0.12.0/qplib_fp.c | 6 +++--- + 2 files changed, 6 insertions(+), 6 deletions(-) + +diff --git a/bnxt_en-1.10.2-220.0.13.0/bnxt.c b/bnxt_en-1.10.2-220.0.13.0/bnxt.c +index b0b99338c425..a2f55dfcd0c3 100644 +--- a/bnxt_en-1.10.2-220.0.13.0/bnxt.c ++++ b/bnxt_en-1.10.2-220.0.13.0/bnxt.c +@@ -10716,7 +10716,7 @@ static void bnxt_free_irq(struct bnxt *bp) + (defined(HAVE_CPUMASK_LOCAL_FIRST) || \ + defined(HAVE_CPUMASK_LOCAL_SPREAD)) + if (irq->have_cpumask) { +- irq_set_affinity_hint(irq->vector, NULL); ++ irq_update_affinity_hint(irq->vector, NULL); + free_cpumask_var(irq->cpu_mask); + irq->have_cpumask = 0; + } +@@ -10787,10 +10787,10 @@ static int bnxt_request_irq(struct bnxt *bp) + break; + } + #endif +- rc = irq_set_affinity_hint(irq->vector, irq->cpu_mask); ++ rc = irq_update_affinity_hint(irq->vector, irq->cpu_mask); + if (rc) { + netdev_warn(bp->dev, +- "Set affinity failed, IRQ = %d\n", ++ "Update affinity failed, IRQ = %d\n", + irq->vector); + break; + } +diff --git a/bnxt_re-220.0.12.0/qplib_fp.c b/bnxt_re-220.0.12.0/qplib_fp.c +index ef5f279fcb79..60d2eda3727d 100644 +--- a/bnxt_re-220.0.12.0/qplib_fp.c ++++ b/bnxt_re-220.0.12.0/qplib_fp.c +@@ -492,7 +492,7 @@ void bnxt_qplib_nq_stop_irq(struct bnxt_qplib_nq *nq, bool kill) + bnxt_qplib_ring_nq_db(&nq->nq_db.dbinfo, res->cctx, false); + /* Sync with last running IRQ handler */ + synchronize_irq(nq->msix_vec); +- irq_set_affinity_hint(nq->msix_vec, NULL); ++ irq_update_affinity_hint(nq->msix_vec, NULL); + free_irq(nq->msix_vec, nq); + + /* Cleanup Tasklet */ +@@ -546,10 +546,10 @@ int bnxt_qplib_nq_start_irq(struct bnxt_qplib_nq *nq, int nq_indx, + + cpumask_clear(&nq->mask); + cpumask_set_cpu(nq_indx, &nq->mask); +- rc = irq_set_affinity_hint(nq->msix_vec, &nq->mask); ++ rc = irq_update_affinity_hint(nq->msix_vec, &nq->mask); + if (rc) + dev_warn(&res->pdev->dev, +- "QPLIB: set affinity failed; vector: %d nq_idx: %d\n", ++ "QPLIB: update affinity failed; vector: %d nq_idx: %d\n", + nq->msix_vec, nq_indx); + nq->requested = true; + bnxt_qplib_ring_nq_db(&nq->nq_db.dbinfo, res->cctx, true); +-- +2.29.2 + diff --git a/kernel-rt/files/kernel-x86_64-rt.config.tis_extra b/kernel-rt/files/kernel-x86_64-rt.config.tis_extra index 5aa58f24..78522c5e 100644 --- a/kernel-rt/files/kernel-x86_64-rt.config.tis_extra +++ b/kernel-rt/files/kernel-x86_64-rt.config.tis_extra @@ -1089,3 +1089,5 @@ CONFIG_GVE=m # upgrade to 5.10.99 # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set # CONFIG_FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION is not set + +# CONFIG_BNXT is not set diff --git a/kernel-std/files/kernel-x86_64.config.tis_extra b/kernel-std/files/kernel-x86_64.config.tis_extra index d1610716..be4879b6 100644 --- a/kernel-std/files/kernel-x86_64.config.tis_extra +++ b/kernel-std/files/kernel-x86_64.config.tis_extra @@ -1597,3 +1597,5 @@ CONFIG_PCI_REALLOC_ENABLE_AUTO=y # upgrade to 5.10.99 # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set # CONFIG_FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION is not set + +# CONFIG_BNXT is not set