The following instructions also include details for enabling an NFS over RDMA server. These are provided as an example only, as the NFS over RDMA server is currently not supported with the UEK R3 kernel.
Install an RDMA device, set up InfiniBand and enable IPoIB.
The Oracle Linux OFED packages are available from the following channels:
Oracle Linux 6:
ol6_x86_64_ofed_UEK
Oracle Linux 7:
ol7_x86_64_UEKR3_OFED20
Check that the RDMA device is working.
#
cat /sys/class/infiniband/
4: ACTIVEdriver_name
/ports/1/statewhere
driver_name
is the RDMA device driver, for examplemlx4_0
.Verify the physical InfiniBand interfaces and links.
Check that the hosts can be contacted through the InfiniBand switch, by using commands such as ibhosts, and ibnetdiscover.
Check the connection between the NFS client and NFS server.
You can configure the settings for an InfiniBand interface in the
/etc/sysconfig/network-scripts/ifcfg-ib
file.N
You can use the ping command to check the connection. For example:
nfs-server$
ip addr add 10.196.0.101/24 dev ib0
nfs-client$ip addr add 10.196.0.102/24 dev ib0
nfs-server$ping 10.196.0.102
nfs-client$ping 10.196.0.101
Install the
nfs-utils
package on the NFS client and server.Configure the NFS shares.
Edit the
/etc/exports
file. Define the directories that the NFS server will make available for clients to mount, using the IPoIB addresses of the clients. For example:/
export_dir
10.196.0.102(fsid=0,rw,async,insecure,no_root_squash) /export_dir
10.196.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)On the NFS server, load the
svcrdma
kernel module and start the NFS service.Oracle Linux 6:
#
modprobe svcrdma
#service nfs start
#echo rdma 20049 > /proc/fs/nfsd/portlist
Oracle Linux 7:
#
modprobe svcrdma
#systemctl start nfs-server
#echo rdma 20049 > /proc/fs/nfsd/portlist
NoteThe
rdma 20049
setting does not persist when the NFS service is restarted. You have to set it each time the NFS service starts.On the NFS client, load the
xprtrdma
kernel module and start the NFS service.#
modprobe xprtrdma
#service nfs start
#mount -o proto=rdma,port=20049
host
:/export
/mntwhere
host
is the host name or IP address of the IPoIB server, andexport
is the name of the NFS share.To check that the mount over RDMA is successful, check the
proto
field for the mount point.#
nfsstat -m
/mnt from 10.196.0.102:/export Flags: rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049, ...Alternatively:
#
cat /proc/mounts