2

I'm trying out RDMA on NFS and noticed that it does not seem to work with NFSv4:

[grant@host ~]$ sudo mount -t nfs -o rdma,proto=rdma,vers=4 10.99.99.98:/ifs/rdma-test /mnt/powerscale_rdma mount.nfs: Protocol family not supported [grant@host ~]$ sudo mount -t nfs -o rdma,proto=rdma,vers=3 10.99.99.98:/ifs/rdma-test /mnt/powerscale_rdma

nfsv3 loads up and runs just fine, but nfsv4 gets you mount.nfs: Protocol family not supported. This answer on NFS4, insecure, port number, rdma contradiction help seems to indicate it could work, but it's not really clear how.

Is there a way to run NFSv4 with RDMA?

Improve this question
2
  • 1
    hm, which network hardware are you using there? (it's a bit surprising!) Commented May 23 at 15:39
  • 1
    Intel E810 on a Dell R7625 which is directly connected to a mellanox card on a PowerScale. Honestly, I'm not sure how to get the exact model off of a PowerScale remotely, but I know it's a connectx card from within the last few years - IE: it should work. IT has the SUPPORTS_RDMA_RRoCE flag when you run isi network interfaces list Commented May 23 at 16:55

2 Answers 2

Reset to default
2

Is there a way to run NFSv4 with RDMA?

yes, I run rdma over mellanox infiniband in rhel-8.10 for nfs vers=4.2. The first thing is the network hardware needs to support rdma - if I try to do rdma over my 1gbps intel NIC will get that or some similar error.

If you are sure your network hardware should do rdma then:

in /etc/nfs.conf you will see

[nfsd] ... # rdma=n # rdma-port=20049

under the nfsd section by default it is off. You need to change that to y and then either reboot or restart all the necessary nfs services so it is in affect.

RDMA in rhel-7.x was slightly problematic, it seems to have been considerably cleaned up and streamlined in rhel-8 {and later}. My experience is everything including rdma works best if your /etc/nfs.conf and /etc/nfsmount.conf is the redhat (or rocky) default which is basically everything commented out; and then simply do rdma=y. When things start changing from default is when problems arise... the server's and client's conf files have to be in sync (i.e. port numbers and protocols) for things to work and it's easy to miss one little thing.

RDMA bypasses the firewall so you should not need to open whatever port in the firewall or disable the firewall (when in doubt easiest troubleshooting if need be is to disable the firewall or course)

make sure no explicit ports or happening for lockd, statd, and so on; just let everything happen over 2049/tcp (which needs to be open in the firewall)

A simple mount -o rdma ip_address:/foldername /foldername is all that's needed in rhel/rocky 8 and later.

your rocky 9.5 is the same as rhel-9.5

Improve this answer
2
  • I'm wondering why, for OP, NFSv3 seems to work. Is it because Linux's RDMA implementation is NFSv4-only and the v3 mount attempt just ignores the option outright? (I probably should reenable NFSv3 here and test.) Commented May 24 at 14:55
  • Just checked. I already have rdma=y and rdma-port=20049. Have confirmed all network hardware supports it both by checking models, the RDMA flags, and the fact it works with v3. Same error persists unfortunately. Commented May 30 at 17:44
1

In my case, I discovered NFSv4 is only supported on Dell OneFS 9.8 and later. I'm on 9.7

See NFS over RDMA overview. I'm going to upgrade and test.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.