1

I am giving myself a crash course on infiniband in RHEL 7.9. I have server(s) each having an HDR ConnectX-6 card installed; this is in my lab and it's on a simple LAN where each server is connected to a Q8700 mlnx switch and for starters we just want basic network functionality over the infiniband so we can ssh, scp and create a basic nfs mount between servers.

  • I have 5 servers, named c1 to c5
  • Each is running RHEL 7.9 x86-64
  • I did mlnxofedinstall from the MLNX_OFED_LINUX-5.2-1.0.4.0-rhel7.9-x86_64.iso on each
  • I did systemctl enable openibd on each followed by a reboot
  • after that I manually /etc/init.d/opensmd start but only on c1
  • using the NetworkManager gui in RHEL 7 I configured the ib0 interface to be
    • 192.168.2.1 to 192.168.2.5 for c1..c5 respectively
    • subnet mask is 255.255.255.0 for each with no gateway specified.
    • if i do ip ib0 it reports link speed of 100000
  • is this all there is to it or am I missing a whole bunch of other stuff ?
  • in doing scp over my copper 1gbps network, I always see a solid 112MB/sec; my take is 1000 mbps / 8 = 125 MBps then at 89.6% efficiency is 112 MBps. If I assume 100000 mb/s on HDR infiniband datagram IPoIB being 100x faster I should see 11200 MB/sec ?
    • I am getting 262 MB/sec currently (scp'ing a 30gb tar file) having done only what's mentioned above. Should I be getting better if so how much?
Improve this question

3 Answers 3

Reset to default
0

Make sure ib_ipoib module is loaded:

 lsmod | grep ipoib ib_ipoib 124872 0 ib_cm 53085 2 rdma_cm,ib_ipoib ib_core 357959 8 rdma_cm,ib_cm,iw_cm,mlx5_ib,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib mlx_compat 55063 13 qedr,rdma_cm,ib_cm,iw_cm,auxiliary,mlx4_ib,mlx5_ib,ib_core,ib_umad,ib_uverbs,mlx5_core,rdma_ucm,ib_ipoib

If it isn't, you can load it via

modprobe ib_ipoib
Improve this answer
0

No, this test will not use your available bandwidth, since the bottleneck is your CPU.

SSH and thus scp is not able to use more than one vCPU.

Improve this answer
0

scp is great, but there is CPU overhead out the wahzoo! setup a simple ftp and do the same test ... you should see significant speed improvements. Unless you have other I/O bottlenecks on your client or host systems.

I typically recommend to use the vendor's official drivers even if they are closed source, rather than the open-source, 3rd party drivers. They can make a HUGE difference with performance on your system.

The PCI bus can still only go so fast however. PCI-X has some different capabilities and limitations. Read the freaking manual and ensure you have your system and network adapter configured correctly!

Do everything right, plan your system configuration smartly and Infiniband will run like a bolt of lightening headed skyward!

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.