If you are using current releases of Solaris 10, Red Hat Enterprise Linux (RHEL), or SUSE Enterprise Linux, the bundled drivers will support basic operation for your Sun Dual-Port 4x InfiniBand (IB) HCA PCI Express (PCIe) ExpressModule (EM) (exceptions noted below). You should not need to update your OS-specific drivers. If you are installing Microsoft Windows Server 2003 on a Sun Blade Server Module, note that it does not include native support for InfiniBand apart from the unreleased Windows Compute Cluster Server Edition. See the following Microsoft site for details about native IB support and the release schedule for Windows Compute Cluster Server Edition:
http://www.microsoft.com/windowsserver2003/default.mspx
This topic includes the following sections:
Installing Drivers
Software Stacks
Related Solaris Documentation
Consult the Sun Blade 8000 Series Product Notes for the most recent information about operating system support and EM driver support.
If you are installing Solaris 10, RHEL, or SUSE software on a blade, you do not need to install or to configure additional drivers for the IB EM. For additional information about the Mellanox MT25208 driver, driver updates, and firmware upgrades, see the following site:
http://www.mellanox.com/support/firmware_table.php
Operating System |
Driver Required |
Patches Required |
Procedure(s) |
---|---|---|---|
Solaris 10 6/06 |
|
None as of this writing. Check SunSolve for updated patch information. |
Use patchadd(1M) and patchrm(1M). |
Red Hat Enterprise Linux Update 3 [64-bit] | |||
SUSE Enterprise Linux SP3 [64-bit] | |||
Windows Server 2003 [32-bit] |
|
None as of this writing. |
See Available InfiniBand Packages for Windows below. |
Windows Server 2003 [64-bit] |
|
None as of this writing. |
See Available InfiniBand Packages for Windows below. |
Exceptions to general behavior and functionality are noted below:
Solaris 10 and InfiniBand EM Hot-plug – Currently, hot-plug functionality is not supported with the Solaris 10 OS.
Solaris 10 Installation and IB EMs – IB EMs cannot be present in either of the EM slots when installing Solaris 10 (by any installation method) to the blade associated with those EM slots. The IB EM must be re-inserted into the EM slot after installation of Solaris 10 to the blade, but before booting Solaris 10 from that blade.
After you have installed Microsoft Windows Server 2003 on the host server blade, you must download and install one of the prerelease Mellanox software packages for Windows 32-bit or Windows 64-bit software from the following site:
http://www.mellanox.com/support/firmware_table.php.
The following table lists the InfiniBand software components contained in the WinIB package.
Component |
Description |
Version |
---|---|---|
IBAL |
Core – IB Access Layer, lower-level driver |
1.2.0 |
IPoIB |
IP communications over IB Fabric |
1.2.0 |
WSD |
WinSocket Direct over IB |
1.2.0 |
SDP |
Socket Driver Protocol over IB |
1.2.0 |
Perf main and vstat |
For verification of low-level performance | |
OpenSM for Windows |
IB-compliant Subnet Manager (SM) and Administrator that runs on top of the OpenIB |
1.8.1 |
Osmtest.exe |
Test application |
The number and maturity of InfiniBand software stacks is improving dramatically. The Mellanox-based IB EM supports multiple software stacks for some operating systems.
Sun currently supports and redistributes the Mellanox IB software stack on the Sun Dual-Port 4x IB EM.
The Mellanox software and documentation are available at the following Sun site:
http://www.sun.com/download/products.xml?id=435959dd
As the popularity of InfiniBand increases, the number of Linux distributions and open source organizations producing drivers and tools will increase. It is prudent to check with your current Linux providers. The following table lists additional Linux-based configurations.
Configuration |
Notes |
---|---|
OpenFabrics.org |
The OpenFabric Alliance (formerly OpenIB) is releasing a Linux stack (Release 1.0) that supports the Sun Dual-Port 4x IB EM. |
Red Hat Enterprise Linux 4 |
RHEL 4 Update 3 distributes a preview version of the OpenFabrics stack (Gen-1). This version is to be used for non-production work. |
SUSE Linux 9 |
SUSE 9 Service Pack 1 offers updates to its preview version of the OpenFabrics stack (Gen-1). This version is to be used for non-production work. |
The InfiniBand software is included with the Solaris 10 operating system. The following figure shows a block diagram of the InfiniBand software stack.
The Solaris InfiniBand stack contains the following modules:
User Direct Access Programming Library (uDAPL)
This library supports service providers written according to the uDAPL 1.2 specification. Sun provides default service provider modules to support the Tavor HCA driver.
IP over InfiniBand Driver (IPoIB)
The IPoIB driver implements the IETF IPoIB protocol and provides IPoIB services for all IB EM ports present in the system. This driver enables the standard Solaris TCP/IP stack to operate over InfiniBand. The link layer driver encapsulates IPv4/6 packets, adds IB multicast support, extends ARP/ND address resolution, and changes the network stack to support link addresses greater than 6 bytes.
InfiniBand Transport Framework (IBTF)
IBTF is the Solaris implementation of the InfiniBand Transport Layer (IBTL). IBTF consists of InfiniBand Device Manager (IBDM), InfiniBand Communications Manager (IBCM), and an InfiniBand Nexus driver. The Nexus driver along with IBDM creates and manages InfiniBand devices. The following devices are supported:
IB I/O controller (IOC) devices
IB pseudo devices
IB service devices
HCA, port, virtual physical point of attachment (VPPA) service devices
See ib(4)
and ib(7d)
for details.
IBCM provides a transport layer abstraction to IB clients to set up reliable connected channels along with service, multicast, and path lookup related
functionality. See ibcm(7d)
for details.
InfiniBand Management Framework (IBMF)
IBMF is a complementary component to IBTF that enables IB clients to provide and access management services over the IB fabrics. It has two logical components. One component deals with sending and receiving management data to any type of management entity. The other component deals with sending and receiving management data to the Subnet Administrator (SA) only. Since every IB client needs to send and receive management data to the SA, IBMF provides functions that specifically handle the SA interaction.
Tavor HCA Driver
The Tavor HCA driver controls the Tavor hardware and enables the transfer of data over the IB wire. This driver interfaces with the IBTF and provides an implementation of Channel Interfaces that are defined by IBTF. It also enables management applications and agents to access the IB fabric.
Related Solaris Documentation
Solaris Manuals
System Administration Guide: Devices and Files System (Sun part number 817Â5093) for information on InfiniBand devices.
System Administration Guide: Network Services (Sun part number 816Â4555) for general information on the InfiniBand network.
Sun Cluster 3.x Hardware Administration Manual for Solaris OS (Sun part number 817Â0168) for information on administering Sun Cluster hardware.
You can view or print Sun documentation at the following web site: http://www.sun.com/documentation.
Solaris Man Pages
cfgadm_ib(1M) – InfiniBand hardwareÂspecific commands for cfgadm
datadm(1M) – maintain DAT static registry file
ifconfig(1M) – configure network interface parameters
libdat(3LIB) – direct access transport library
ib(4) – InfiniBand device driver configuration files
ibmf(7) – InfiniBand Management Transport Framework
daplt(7D) – Tavor uDAPL service driver
ib(7D) – InfiniBand (IB) Bus Nexus Driver
ibcm(7D) – InfiniBand (IB) Communications Manager
ibdm(7D) – Solaris InfiniBand (IB) device manager
tavor(7D) – InfiniBand (IB) Tavor driver
Sun currently supports and redistributes the Mellanox WinIB software stack on the Sun Dual-Port 4x IB EM.
The Mellanox software and documentation are available at the following Sun site: