1.4.1 Using the NFS over RDMA Client

The following instructions also include details for enabling an NFS over RDMA server. These are provided as an example only, as the NFS over RDMA server is currently not supported with the UEK R3 kernel.

Install an RDMA device, set up InfiniBand and enable IPoIB.
The Oracle Linux OFED packages are available from the following channels:
- Oracle Linux 6: ol6_x86_64_ofed_UEK
- Oracle Linux 7: ol7_x86_64_UEKR3_OFED20
Check that the RDMA device is working.
```
# cat /sys/class/infiniband/driver_name/ports/1/state
4: ACTIVE
```
where driver_name is the RDMA device driver, for example mlx4_0.
Verify the physical InfiniBand interfaces and links.
Check that the hosts can be contacted through the InfiniBand switch, by using commands such as ibhosts, and ibnetdiscover.
Check the connection between the NFS client and NFS server.
You can configure the settings for an InfiniBand interface in the
/etc/sysconfig/network-scripts/ifcfg-ibN file.
You can use the ping command to check the connection. For example:
```
nfs-server$ ip addr add 10.196.0.101/24 dev ib0
nfs-client$ ip addr add 10.196.0.102/24 dev ib0
nfs-server$ ping 10.196.0.102 
nfs-client$ ping 10.196.0.101
```
Install the nfs-utils package on the NFS client and server.
Configure the NFS shares.
Edit the /etc/exports file. Define the directories that the NFS server will make available for clients to mount, using the IPoIB addresses of the clients. For example:
```
/export_dir 10.196.0.102(fsid=0,rw,async,insecure,no_root_squash) 
/export_dir 10.196.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
```
On the NFS server, load the svcrdma kernel module and start the NFS service.
Oracle Linux 6:
```
# modprobe svcrdma
# service nfs start
# echo rdma 20049 > /proc/fs/nfsd/portlist
```
Oracle Linux 7:
```
# modprobe svcrdma
# systemctl start nfs-server
# echo rdma 20049 > /proc/fs/nfsd/portlist
```
Note
The rdma 20049 setting does not persist when the NFS service is restarted. You have to set it each time the NFS service starts.
On the NFS client, load the xprtrdma kernel module and start the NFS service.
```
# modprobe xprtrdma
# service nfs start
# mount -o proto=rdma,port=20049 host:/export /mnt 
```
where host is the host name or IP address of the IPoIB server, and export is the name of the NFS share.
To check that the mount over RDMA is successful, check the proto field for the mount point.
```
# nfsstat -m
/mnt from 10.196.0.102:/export
Flags: rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,
...
```
Alternatively:
```
# cat /proc/mounts
```

Known Issues

Any mounted file systems must be unmounted on the NFS client before you shut down the NFS server. Otherwise the NFS server hangs when you shut down.