Introduction to the Sun Blade 6048 Switched InfiniBand Network ExpressModule

C H A P T E R 1

This chapter provides an overview of the Sun Blade 6048 Switched InfiniBand Network ExpressModule (IB NEM). This chapter also lists the host platforms and operating systems that support the NEM.

This chapter includes the following sections:

Section 1.1, Product Features

Section 1.2, Platform and Operating System Support

Section 1.3, IB NEM Indicators, Buttons, and Ports

You can order additional Sun Blade 6048 Switched InfiniBand Network ExpressModules from Sun Microsystems using the following Marketing part number: X5196A-Z.

1.1 Product Features

The Sun Blade 6048 Switched InfiniBand Network ExpressModule is a 24-port Network ExpressModule that provides two InfiniBand SDR connections to each Sun Blade Server Module. Each 4X SDR port supports an x8 PCI Express (PCIe) connection to its Server Module, providing full-duplex data transfers of up to 20 Gbps.

The NEM supports 24 independent InfiniBand Host Channel Adapter (HCA) ports operating at up to 20-Gbps (DDR) speed, and 12Gigabit (1 Gbps) Ethernet ports. The NEM is also backward compatible with 10-Gbps (SDR) devices.

To connect to the external InfiniBand network, standard InfiniBand connectors (iPASS⁾are usedwith passive copper cables.

TABLE 1-1 lists additional features of the Sun IB NEM.

TABLE 1-1 Sun IB NEM Features
Feature	Description
Hot-plug operations supported	Enables system administrators to easily add or remove IB NEMs, as needed, without powering down the system
IB transfer rate (maximum)	20 Gbps (DDR) per server 480 Gbps aggregate throughput
InfiniBand Trade Association (IBTA) interoperability	Version 1.2
IB uplink interface	iPASS connectors
Host interface	12 PCIe x8 links
Ethernet interface	12 Gigabit ports
PCIe transfer rate (maximum) per lane	2.5 Gbps (PCIe Generation One) 5 Gbps (PCIe Generation Two)
IB interface chip	Mellanox ConnectX IB
Onboard memory	Two 2-MB flash ROM per HCA
Maximum IB cable length	15 meters, using passive copper cables 16 meters, using active copper cables
LED indicators	One green LED per 4x port showing physical link state Attention push button, LED (white), OK LED (green), Service Action Required LED (amber), Ready-to-Remove LED (blue)
Form factor	Sun proprietary form factor
RoHS	5 of 6 compliant
Power consumption	Approximately 240 watts total
Protection	Protected from electrostatic discharge (ESD) and handling damage

1.2 Platform and Operating System Support

This section provides information about selected platforms that are compatible with the heterogeneous InfiniBand network design.

TABLE 1-2 lists operating system support for the IB NEM on the Sun Blade 6000 Series Modular System, and required drivers.

TABLE 1-2 Operating System Support
Operating System	Version Details	Driver Required
CentOS RHEL	Release 4.4 (Final) RHEL4 u4 RHEL4 u5 RHEL5 u1	OFED-1.2.5 or later

For more information about the InfiniBand software stack and related topics, see Chapter 3.

1.3 IB NEM Indicators, Buttons, and Ports

The Sun Blade 6048 Switched InfiniBand Network ExpressModule provides a standard set of status indicators, buttons, and ports on the back panel. TABLE 1-3 describes the components indicated by the numbers.

FIGURE 1-1 Sun Blade 6048 Switched InfiniBand Express NEM Back Panel Indicators, Buttons, and Ports

TABLE 1-3 Sun Blade 6048 Switched InfiniBand Express NEM Back Panel Components and Functions
Number	Component Name	Color	Description
1	Ejector lever		To remove the NEM, you open this ejector lever.
2	Ready-to-Remove indicator	Blue	Provides the following indications: Steady On - Lights when it is safe to remove the associated NEM from the NEM slot in the chassis. Off - The NEM is not ready for removal. This indicator is normally lit from the ILOM web interface through the Chassis Monitoring Module (CMM) or by pressing the Attention button.
3	Service Action Required indicator	Amber	Provides the following indications: Steady On - Lights when there is a fault associated with the NEM. Off - The NEM has no fault condition.
4	OK indicator	Green	Displays the different states of module initialization. Provides the following indications: Steady On - Lights steadily when NEM is operating normally. Slow Blink - Blinks slowly when NEM is transitioning from one state to the next. Standby Blink - Blinks more slowly when NEM has been inserted and NEM is waiting for the Attention button to be pressed. Off - NEM has no power or one of the other LEDs is lit. The primary purpose of this OK indicator is to provide a visual cue to the operator as to what will happen when using the Attention button.
5	Attention button		When pressed, the Attention button enables you to activate (if inserting) or prepare to remove (if removing) a NEM during a hot-plug operation.
6	Locate indicator	White	The Locate indicator is a bright white button with an associated indicator that blinks (4 Hz) when initiated (remotely) from the ILOM web interface. The Locate indicator enables a system operator to easily locate a NEM in a system within a large data center. Once activated, this indicator will time-out after 30 minutes. When pressed, the Locate button lights the Locate indicator. The Locate indicator provides these indications: Fast blink - Identifies a specific NEM in the chassis. Lights when the LED is initiated from the web interface remotely or from a press of the Locate button locally. Off - NEM locator function has not been selected.
7	Connectors		The iPASS connector combines three 4x links per connector.
8	Connectors		RJ-45 Ethernet connector.

Each InfiniBand port has one LED indicator, as shown in FIGURE 1-2. TABLE 1-4 lists and describes the LED indicator.

FIGURE 1-2 Sun Blade 6048 Switched InfiniBand Express NEM Port LED Indicators

TABLE 1-4 InfiniBand Port LED Indicator Description
Number	Color	Description
1	Green	This physical link LED illuminates when the port is electrically active, that is, when a driver is attached and a physical link to a remote switch (or, possibly an HCA) has been established.