When using RHEL 8.8, there is a choice to be made for users with Mellanox InfiniBand hardware. They can either utilize Red Hat's included InfiniBand Support group installation (which I choose to do during the installation process) in order to enable InfiniBand networking, or they can opt for MLNX_OFED_LINUX-23.04-1.1.3.0-rhel8.8-x86_64.iso and execute their mlnxofedinstall script, which replaces Red Hat packages with their own.
Aside from firmware updates, does anyone know of any advantages of using the MLNX path as opposed to the Red Hat path for setting up InfiniBand?
The MLNX approach generally involves more setup and configuration, so I am interested in any performance justifications for using the MLNX method (considering that I am using their hardware) instead of what Red Hat provides.
For reference:
Usage: ./mlnxofedinstall [OPTIONS] Installation control: --force Force installation --tmpdir Change tmp directory. Default: /tmp -k|--kernel <version> Default on this system: 3.10.0-1160.92.1.el7.x86_64 -s|--kernel-sources <path> Default on this system: /lib/modules/3.10.0-1160.92.1.el7.x86_64/build -U|--update Update installed version --with-32bit Install 32-bit libraries. Relevant for x86_64 platforms --without-32bit Skip 32-bit libraries installation (Default). Relevant for x86_64 platforms --distro Set Distro name for the running OS (e.g: rhel6.5, sles12sp3). Default: Use auto-detection --skip-distro-check Do not check MLNX_OFED_LINUX vs Distro matching --without-depcheck Run the installation without verifying that all required Distro's packages are installed --check-deps-only Check for missing required Distro's packages and exit --print-distro Print distribution name and exit --add-kernel-support Add kernel support (Run mlnx_add_kernel_support.sh) and install it --add-kernel-support-build-only Add kernel support (Run mlnx_add_kernel_support.sh) and exit --kernel-extra-args '<args>' pass <args> to kernel configure script (single paraeter, space separated) - Pass it along with --add-kernel-support --kmp Install kernel RPMs with KMP support if available - Enabled by default with official MLNX_OFED_LINUX releases - Disabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh - Passing it along with --add-kernel-support will pass --kmp to mlnx_add_kernel_support.sh --disable-kmp Install kernel RPMs without KMP support if available - Enabled by default if new MLNX_OFED_LINUX was created with mlnx_add_kernel_support.sh --skip-kmp-verify Do not check if the KMP rpms are compatiable with the target kernel --speed-up-kmp Speed up KMP RPMs installation by skipping %post section and running the required operations once for all KMPs --umad-dev-rw Grant non root users read/write permission for umad devices instead of default --umad-dev-na Prevent from non root users read/write access for umad devices. Overrides '--umad-dev-rw' --enable-mlnx_tune Enable Running the mlnx_tune utility --enable-affinity Run mlnx_affinity script upon boot --disable-affinity Disable mlnx_affinity script (Default) --skip-unsupported-devices-check Don't abort if system has an older, unsupported card --enable-opensm Run opensm upon boot --package-install-options RPM install options to use when installing RPM packages (comma separated list) --pre-install-<package> <path to script> Run given script before given package's install --post-install-<package> <path to script> Run given script after given package's install Firmware update control: --without-fw-update Skip firmware update --fw-update-only Update firmware. Skip driver installation --force-fw-update Force firmware update --fw-image-dir Firmware images directory to use instead of default package content Package selection: -c|--config <packages config_file> Example of the config file can be found under docs --all Install all available packages --bluefield Install BlueField packages --hpc Install minimum packages required for HPC --basic Install minimum packages for basic functionality --dpdk Install minimum packages required for DPDK --ovs-dpdk Install DPDK and OVS packages --vma Install minimum packages required for VMA --xlio Install minimum packages required for XLIO --guest Install minimum packages required by guest OS --hypervisor Install minimum packages required by hypervisor OS --with-fabric-collector Enable installing fabric-collector package --with-nvmf Enable NVMEoF support --with-nfsrdma Enable NFSoRDMA support --without-nfsrdma Disable NFSoRDMA support (Default) Extra package filtering: --kernel-only Install kernel space packages only --user-space-only Filter selected packages and install only User Space packages --without-<package> Do not install package --with-<package> Force installing package Miscellaneous: -h|--help Display this help message and exit --post-start-delay <sec> Set openibd POST_START_DELAY parameter in seconds. (Default 0) -p|--print-available Print available packages for current platform And create corresponding ofed.conf file --copy-ifnames-udev Copy compatibility udev rules for interface names Output control: -v|-vv|-vvv Set verbosity level -q Set quiet - no messages will be printed 
