C H A P T E R  1

Product Overview

This chapter provides an overview of the Suntrademark InfiniBand Dual Port 4x DDR PCIe® Fabric Expansion Module (IB-FEM).

This chapter includes the following sections:


Product Description

The Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Module (IB-FEM) is a single-wide InfiniBand DDR Fabric Expansion Module (FEM) based on Mellanox MT25408 ConnectX ASIC. The IB-FEM mezzanine card is installed on a compute blade to provide dual IB connectivity to the Sun Bladetrademark 6048 QDR Switched Network ExpressModule (IB-QNEM). The IB-FEM consists of one Amphenol NexLev connector and one Mellanox ConnectX ASIC chip. The NexLev connector contains a x8 PCIe and two DDR IB interfaces. This module is RoHS compliant. The IB-FEM is shown in FIGURE 1-1.



Note - The Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Module is based on Mellanox Technologies’ MT25408 ConnectX IB adapter (HCA) device.



FIGURE 1-1 Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Module

Figure shows the Fabric Expansion Module


Hardware and Software Requirements

Before installing the IB-FEM, ensure that your system meets the hardware and software requirements in TABLE 1-1.


TABLE 1-1 Hardware and Software Requirements for Sun x64 Servers

Requirements

Hardware or Software

Operating systems

  • RedHat Enterprise Linux (RHEL) 5.3 and 4U7
  • Suse Linux Enterprise Server (SLES) 10 update 2
  • CentOS 5.3

Other software

  • OpenFabrics Enterprise Distribution (OFED) 1.4 or later
  • Mellanox OpenFabrics Enterprise Distribution (MLNX_OFED)

Network Express Module

Sun Blade 6048 InfiniBand QDR Switched Network Express Module (IB-QNEM)

Sun Blade Server Module

Sun Blade 6270 Server Module or other compatible server module


Dual Port Connectivity

On some Sun Blade server modules, one port on the IB-FEM is disabled for compatibility with the IB-QNEM. Therefore two IB-FEMs are required for dual port connectivity. The Sun Blade X6270 Server Module requires that two IB-FEMs be installed on the server module for dual port connectivity to the IB-QNEM.


Interfaces

InfiniBand Interface

The IB-FEM is compliant with the InfiniBand Architecture Specification, Release 1.2. The IB-FEM has two compliant 4X InfiniBand ports, A and B. The IB-FEM provides access to these ports from the Sun Blade 6048 InfiniBand QDR Switched Network Express Module (IB-QNEM) through the chassis midplane. The IB-QNEM provides the InfiniBand connectors for external InfiniBand cables that are also compliant with the InfiniBand Architecture Specification, Release 1.2.

PCI Express Interface

The Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Module has eight Tx/Rx pairs of SerDes, providing for a PCI Express x8 interface, version 2.0 compatible. The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations.


Memory

The IB-FEM support multiple memory devices through the PCI Express, Flash, and I2C compatible interfaces.

System Memory

The IB-FEM uses the PCI Express interface to store and access IB fabric connection information on the system memory.


Regulatory Compliance

Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Modules (also referred to as host channel adapters or HCAs) are components of the Sun server module in which they installed. See the Agency certifications of the applicable server module to determine the relevant Agency approvals for the server module and HCA combination.

To comply with legal EMI emissions regulations, the Sun Blade 6048 Modular Server must be provided with a front door assembly when used with the Sun InfiniBand Dual Port 4x DDR PCIe Fabric Expansion Module. The front door should remain closed during normal operation and opened only during product servicing or when power is removed from the product.