Sun Blade 6048 Switched InfiniBand Network ExpressModule User’s Guide
|
|
Introduction to the Sun Blade 6048 Switched InfiniBand Network ExpressModule
|
This chapter provides an overview of the Sun Blade 6048 Switched InfiniBand Network ExpressModule (IB NEM). This chapter also lists the host platforms and operating systems that support the NEM.
This chapter includes the following sections:
You can order additional Sun Blade 6048 Switched InfiniBand Network ExpressModules from Sun Microsystems using the following Marketing part number: X5196A-Z.
1.1 Product Features
The Sun Blade 6048 Switched InfiniBand Network ExpressModule is a 24-port Network ExpressModule that provides two InfiniBand SDR connections to each Sun Blade Server Module. Each 4X SDR port supports an x8 PCI Express (PCIe) connection to its Server Module, providing full-duplex data transfers of up to 20 Gbps.
The NEM supports 24 independent InfiniBand Host Channel Adapter (HCA) ports operating at up to 20-Gbps (DDR) speed, and 12Gigabit (1 Gbps) Ethernet ports. The NEM is also backward compatible with 10-Gbps (SDR) devices.
To connect to the external InfiniBand network, standard InfiniBand connectors (iPASS) are used with passive copper cables.
TABLE 1-1 lists additional features of the Sun IB NEM.
TABLE 1-1 Sun IB NEM Features
Feature
|
Description
|
Hot-plug operations supported
|
Enables system administrators to easily add or remove IB NEMs, as needed, without powering down the system
|
IB transfer rate (maximum)
|
20 Gbps (DDR) per server
480 Gbps aggregate throughput
|
InfiniBand Trade Association (IBTA) interoperability
|
Version 1.2
|
IB uplink interface
|
iPASS connectors
|
Host interface
|
12 PCIe x8 links
|
Ethernet interface
|
12 Gigabit ports
|
PCIe transfer rate (maximum)
per lane
|
2.5 Gbps (PCIe Generation One)
5 Gbps (PCIe Generation Two)
|
IB interface chip
|
Mellanox ConnectX IB
|
Onboard memory
|
Two 2-MB flash ROM per HCA
|
Maximum IB cable length
|
15 meters, using passive copper cables
16 meters, using active copper cables
|
LED indicators
|
- One green LED per 4x port showing physical link state
- Attention push button, LED (white), OK LED (green), Service Action Required LED (amber), Ready-to-Remove LED (blue)
|
Form factor
|
Sun proprietary form factor
|
RoHS
|
5 of 6 compliant
|
Power consumption
|
Approximately 240 watts total
|
Protection
|
Protected from electrostatic discharge (ESD) and handling damage
|
1.2 Platform and Operating System Support
This section provides information about selected platforms that are compatible with the heterogeneous InfiniBand network design.
TABLE 1-2 lists operating system support for the IB NEM on the Sun Blade 6000 Series Modular System, and required drivers.
TABLE 1-2 Operating System Support
Operating System
|
Version Details
|
Driver Required
|
CentOS
RHEL
|
Release 4.4 (Final)
RHEL4 u4
RHEL4 u5
RHEL5 u1
|
OFED-1.2.5 or later
|
For more information about the InfiniBand software stack and related topics, see Chapter 3.
1.3 IB NEM Indicators, Buttons, and Ports
The Sun Blade 6048 Switched InfiniBand Network ExpressModule provides a standard set of status indicators, buttons, and ports on the back panel. TABLE 1-3 describes the components indicated by the numbers.
FIGURE 1-1 Sun Blade 6048 Switched InfiniBand Express NEM Back Panel Indicators, Buttons, and Ports
TABLE 1-3 Sun Blade 6048 Switched InfiniBand Express NEM Back Panel Components and Functions
Number
|
Component Name
|
Color
|
Description
|
1
|
Ejector lever
|
|
To remove the NEM, you open this ejector lever.
|
2
|
Ready-to-Remove indicator
|
Blue
|
Provides the following indications:
- Steady On - Lights when it is safe to remove the associated NEM from the NEM slot in the chassis.
- Off - The NEM is not ready for removal.
This indicator is normally lit from the ILOM web interface through the Chassis Monitoring Module (CMM) or by pressing the Attention button.
|
3
|
Service Action Required indicator
|
Amber
|
Provides the following indications:
- Steady On - Lights when there is a fault associated with the NEM.
- Off - The NEM has no fault condition.
|
4
|
OK indicator
|
Green
|
Displays the different states of module initialization.
Provides the following indications:
- Steady On - Lights steadily when NEM is operating normally.
- Slow Blink - Blinks slowly when NEM is transitioning from one state to the next.
- Standby Blink - Blinks more slowly when NEM has been inserted and NEM is waiting for the Attention button to be pressed.
- Off - NEM has no power or one of the other LEDs is lit.
The primary purpose of this OK indicator is to provide a visual cue to the operator as to what will happen when using the Attention button.
|
5
|
Attention button
|
|
When pressed, the Attention button enables you to activate (if inserting) or prepare to remove (if removing) a NEM during a hot-plug operation.
|
6
|
Locate indicator
|
White
|
The Locate indicator is a bright white button with an associated indicator that blinks (4 Hz) when initiated (remotely) from the ILOM web interface.
The Locate indicator enables a system operator to easily locate a NEM in a system within a large data center. Once activated, this indicator will time-out after 30 minutes.
When pressed, the Locate button lights the Locate indicator.
The Locate indicator provides these indications:
- Fast blink - Identifies a specific NEM in the chassis. Lights when the LED is initiated from the web interface remotely or from a press of the Locate button locally.
- Off - NEM locator function has not been selected.
|
7
|
Connectors
|
|
The iPASS connector combines three 4x links per connector.
|
8
|
Connectors
|
|
RJ-45 Ethernet connector.
|
Each InfiniBand port has one LED indicator, as shown in FIGURE 1-2. TABLE 1-4 lists and describes the LED indicator.
FIGURE 1-2 Sun Blade 6048 Switched InfiniBand Express NEM Port LED Indicators
TABLE 1-4 InfiniBand Port LED Indicator Description
Number
|
Color
|
Description
|
1
|
Green
|
This physical link LED illuminates when the port is electrically active, that is, when a driver is attached and a physical link to a remote switch (or, possibly an HCA) has been established.
|
Sun Blade 6048 Switched InfiniBand Network ExpressModule User’s Guide
|
820-2189-10
|
|
Copyright © 2008, Sun Microsystems, Inc. All Rights Reserved.