InfiniBand (IB) is an I/O technology based on switched fabrics. It provides a high bandwidth and low latency interconnect for attaching I/O devices to hosts and for host-to-host communication. IB devices are managed by the Oracle Solaris IB nexus driver.
Oracle Solaris supports the following devices:
IP over IB (IPoIB) devices – Enables transporting IP packets over IB connections. This feature is implemented by the ibp driver. For more information, see the ibp(4D) man page.
Ethernet over IB (EoIB) – Enables transporting Ethernet frames over IB fabric.
Socket Direct Protocol (SDP) – Supports sockets over IB.
Reliable Datagram Service version 3 (RDSv3) and Reliable Datagram Service (RDS).
NFS over Remote Direct Memory Access (NFSoRDMA) – Provides NFS services over IB using RDMA.
iSCSI Extension for RDMA (iSER) – Provides RDMA data transfer capability to the iSCSI protocol.
Open Fabric User Verb (OFUV).
Oracle Solaris now supports PCIe scalable system interface (PSIF) host channel adapter (HCA). For more information about PSIF, see the product notes in Oracle Dual Port QDR InfiniBand Adapter M4 Documentation Library.
The IB nexus driver queries the Oracle Solaris IB Device Manager (IBDM) for communication services to enumerate the IB Port, HCA_SVC, and IB VPPA (virtual physical point of attachment) devices.
The IB partition link used for data transfers represents a new part class of data link that you manage by using dladm subcommands. You can create one IB partition link per each partition key (P_Key) on the port on top of an IB physical link.
Port devices bind a communication service to a specific port of a Host Channel Adapter (HCA). HCA_SVC devices bind a communication service to a specific HCA. VPPA devices bind a communication service to a combination of a port and a partition key (P_key). Note that port devices and HCA_SVC devices always use a partition key whose value is zero. Port, HCA_SVC, and VPPA devices are children of the HCA and are enumerated in the ib.conf file. For more information, see the ib(4D) man page.
Input Output Controller (IOC) devices are children of the IB nexus driver and are part of an I/O unit. Pseudo devices are also children of the IB nexus driver. This term refers to all other devices that provide their own configuration files to enumerate. For more information, see the ib(5) man page.
The possible IB device tree path names are as follows:
IOC device – /ib/ioc@1730000007F510C,173000007F50
IB pseudo device – /ib/driver@unit-address
IB VPPA device – /pci@1f,2000/pci@1/pci15b3,5a44@0/ibport@,port#,P_key,service
IB HCA_SVC device –
IB Port device – /pci@1f,2000/pci@1/pci15b3,5a44@0/ibport@<port#>,0,service
HCA – /pci@1f,2000/pci@1/pci15b3,5a44@0
The IB components that are part of the path names are:
A communication service. For example, ipib is the communication service used by the ibd kernel client driver.
The partition link key value.
The port number.
The IB kernel client driver's unit address as specified in its driver.conf file. For more information, see the driver.conf(5) man page.
EoIB uses the services provided by the Oracle Solaris IB framework. However, EoIB is not a child of the IB nexus driver. Oracle Solaris EoIB is a single-instance pseudo device driver with a device tree path of /pseudo/eib@0. An EoIB datalink is a special eoib class of virtual links that is created and managed by dladm subcommands. You can create one EoIB datalink over an existing IB physical link for each EoIB Gateway discovered on the IB fabric from that IB physical link. To display a list of discovered EoIB Gateways, use the dladm subcommands.
For information about using IB diagnostic commands and utilities, see Monitoring and Troubleshooting IB Devices.
The IB related software packages are as follows:
driver/infiniband/connectx – Mellanox ConnectX Family InfiniBand HCA and 10GbE NIC drivers
system/io/infiniband/ethernet-over-ib – Network device driver implementing Ethernet over InfiniBand
system/io/infiniband/ib-device-mgt-agent – InfiniBand Device Manager Agent
system/io/infiniband/ib-sockets-direct – InfiniBand layered Sockets Direct Protocol
system/io/infiniband/ip-over-ib – Network device driver supporting the IP over InfiniBand (IPoIB) protocol
system/io/infiniband/open-fabrics – Open Fabrics kernel components
system/io/infiniband/reliable-datagram-sockets-v3 – Reliable Datagram Sockets (RDSv3)
system/io/infiniband/reliable-datagram-sockets – Reliable Datagram Sockets
system/io/infiniband/rpc-over-rdma – InfiniBand RPC over RDMA Driver
system/io/infiniband – InfiniBand Framework
driver/infiniband/sif – Oracle PCIe Scalable System Interface (PSIF) InfiniBand HCA driver