NFS Over RDMA

Starting in the Oracle Solaris 11.1 release, the default transport for NFS is the Remote Direct Memory Access (RDMA) protocol. This protocol provides memory-to-memory transfer of data over high-speed networks. Specifically, RDMA provides remote data transfer directly to and from memory without CPU intervention. RDMA also provides direct data placement, which eliminates data copies and therefore further eliminates CPU intervention. Thus, RDMA frees up not only the host CPU, but also reduces contention for the host memory and I/O buses. To provide this capability, RDMA combines the interconnect I/O technology of InfiniBand, which you can use on both SPARC and x86 platforms, with the Oracle Solaris operating system. The following figure shows the relationship of RDMA to other protocols, such as UDP and TCP.

Figure 2-1  Relationship of RDMA to Other Protocols

Because RDMA is the default transport protocol for NFS, no special share or mount options are required to use RDMA on a client or server. The existing automounter maps, vfstab and file system shares, work with the RDMA transport. NFS mounts over the RDMA transport occur transparently when InfiniBand connectivity exists between the client and the server. InfiniBand connectivity feature works on both SPARC and x86 platforms. If the RDMA transport is not available on both the client and the server, the TCP transport is the initial fallback, followed by UDP if TCP is unavailable. However, if you use the –proto=rdma mount option, NFS mounts are forced to use RDMA only.

To specify the use of only TCP and UDP, you can use the –proto=tcp/udp mount option. This option disables RDMA on an NFS client. For more information about NFS mount options, see the mount_nfs (1M) and mount (1M) man pages.