JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Sun Datacenter InfiniBand Switch 648 Topic Set
search filter icon
search icon

Document Information

Preface

Related Documentation

Documentation, Support, and Training

Third-Party Web Sites

Sun Welcomes Your Comments

Administration

Administering the Switch

Understanding Administrative Commands

Monitoring the Switch

Monitoring the InfiniBand Fabric

Controlling the Switch

Controlling the InfiniBand Fabric

Diagnosing a Problem

Identifying the Problem

Troubleshooting the Switch

Understanding Signal Routing Through the Switch

Understanding Switch Startup and Component Addition

Switch GUIDs Overview

Remote Management

Understanding ILOM on the Switch

ILOM Documentation

ILOM Overview

Supported Features

Understanding ILOM Targets

Installing the ILOM Firmware

Firmware Layout

rupgrade_tool Application

Download the ILOM Firmware

Install the ILOM Firmware

Administering ILOM (CLI)

CLI Overview

Access ILOM From the CLI

Switching Between the ILOM Shell and the Linux Shell

Monitoring ILOM Targets (CLI)

Controlling ILOM Targets (CLI)

Upgrading the Switch Firmware Through ILOM (CLI)

Administering ILOM (Web Interface)

Web Interface Overview

Access ILOM From the Web Interface

Monitoring ILOM Targets (Web Interface)

Controlling ILOM Targets (Web Interface)

Upgrading the Switch Firmware Through ILOM (Web Interface)

Understanding ILOM Commands

cd Command

create Command

delete Command

dump Command

exit Command

help Command

load Command

set Command

show Command

version Command

Installation

Preparing the Site

Routing Service Cables

Understanding InfiniBand Cabling

Understanding Switch Specifications

Installing the Switch

Understanding the Installation

Supported Configurations

Fabric Card Distribution

Line Card Distribution

Fabric Card and Line Card Fillers

Installation Preparation

Suggested Tools for Installation

Antistatic Precautions for Installation

Installation Responsibilities

Installer Experience

Installation Sequence

Installing the InfiniBand Software Stack

Install the HPC Software Suite

Unpacking the Chassis

Remove the Shipping Carton Contents

Unsecure the Chassis From the Pallet

Preparing the Chassis

Install the Rail Assemblies in the Rack

Install the Cable Trees

Install the Cable Curves

Install the Chassis in the Rack

Inspecting the Components

Inspecting the Midplane Condition

Midplane Fabric Card Connector Nomenclature

Inspect the Pins on the Rear Side of the Midplane

Midplane Line Card Connector Nomenclature

Inspect the Pins on the Front Side of the Midplane

Inspecting the Fabric Cards Condition

Fabric Card Features

Fabric Card Handling Guidelines

Inspect the Fabric Card Chassis Condition

Inspect the Fabric Card Retainer Bolts Condition

Inspect the Fans Condition

Inspect the Fabric Card XBOW Connectors Condition

Inspecting the Line Cards Condition

Line Card Features

Line Card Handling Guidelines

Inspect the Line Card Chassis Condition

Inspect the Line Card Retainer Bolts Condition

Inspect the Line Card XBOW Connectors Condition

Inspect the Line Card CXP Connectors Condition

Inspecting the InfiniBand Cables Condition

InfiniBand Cable Features

Inspect the InfiniBand Cables Hardware Condition

Inspect the InfiniBand Cables Connector Condition

Installing the Components

Install the Midplane Stiffener

Install the Fabric Cards or Fabric Card Fillers

Remove the Midplane Stiffener

Install the Line Cards

Powering On the CMCs

Power Supply Addresses

Power Supply Status LEDs

CMC Addresses

CMC Status LEDs

Attach Management Cables

Attach Power Cords

Accessing the CMC

Access the CMC From the Serial Management Port

Set the Initial UBoot Parameters

Access the CMC From the Network Management Port

Verify the CMC Status

Display the CMC FRU ID Information

Verify the Power Supply Status

Display the Power Supply FRU ID Information

Powering On the Fabric Cards

Fabric Card Addresses

Fabric Card Status LEDs

Enable the Fabric Cards

Create the guid.txt File for Installation

Power On the Fabric Cards

Verify the Fabric Card Status

Display the Fabric Card FRU ID Information

Powering On the Line Cards

Line Card Addresses

Line Card Status LEDs

Enable the Line Cards

Power On the Line Cards

Verify the Line Card Status

Display the Line Card FRU ID Information

Connecting InfiniBand Cables

CXP Connector Addresses

CXP Connector Status LEDs

Attach Cables to the CXP Connectors

Check Link Status

Display the InfiniBand Cable FRU ID Information

Verifying the InfiniBand Fabric

Discover the InfiniBand Fabric Topology

Perform Diagnostics on the InfiniBand Fabric

Validate the InfiniBand Fabric and Report Errors

Service

Servicing the Switch

Understanding Service Procedures

Servicing the Power Supplies

Servicing the CMCs

Servicing the Fans

Servicing the Fabric Cards and Fabric Card Fillers

Servicing the InfiniBand Cables

Servicing the Line Cards

Doing Supportive Software Tasks

Reference

Command Reference

Understanding Switch-Specific Commands

Understanding CLIA Commands

Understanding InfiniBand Software Commands

Index

Create the guid.txt File for Installation

If your InfiniBand fabric uses the Fat-Tree routing algorithm, you must provide a modified listing of fabric card GUIDs, called the root node GUID file, for the Subnet Manager. The /etc/opensm/guid.txt file is such a file. This procedure describes how to create the guid.txt file.


Note - In this procedure, you issue commands to both the CMC and the Linux InfiniBand host.


  1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

    See the table in Installation Sequence.

  2. Type the following command on the CMC to retrieve the fabric card base GUID:
    # getbaseguid fc slot

    where slot is the number of the fabric card (08). For example:

    # getbaseguid fc 0
    Base GUID = 0x00bad0fc0009f000
    #
  3. Record the GUID to a text editor on your Linux InfiniBand host.
  4. Repeat Step 2 and Step 3 for each fabric card.

    Once you have recorded each fabric card’s GUID, you will have a list similar to the following on your Linux InfiniBand host:

    0x00bad0fc0009f000
    0x00bad0fc0033f100
    0x00bad00017fcf200
    0x00bad00005fcf300
    0x00bad00003fcf400
    0x00bad00007fcf500
    0x00bad0fc0011f600
    0x00bad0b00002f700
    0x00bad0b00001f800

    Note - If you have fewer than 9 fabric cards, your list will be smaller.


  5. For each GUID, add the following values to make two new sums:
    • 0xa2

    • 0xb2


    Note - For GUIDs that end with two zeros (00), you are effectively replacing the zeros with a2 and b2.


    For example, 0x00bad0fc0009f000 becomes:

    • 0x00bad0fc0009f0a2

    • 0x00bad0fc0009f0b2

  6. Repopulate your list with the modified GUIDs.

    For example:

    0x00bad0fc0009f0a2
    0x00bad0fc0009f0b2
    0x00bad0fc0033f1a2
    0x00bad0fc0033f1b2
    0x00bad00017fcf2a2
    0x00bad00017fcf2b2
    0x00bad00005fcf3a2
    0x00bad00005fcf3b2
    0x00bad00003fcf4a2
    0x00bad00003fcf4b2
    0x00bad00007fcf5a2
    0x00bad00007fcf5b2
    0x00bad0fc0011f6a2
    0x00bad0fc0011f6b2
    0x00bad0b00002f7a2
    0x00bad0b00002f7b2
    0x00bad0b00001f8a2
    0x00bad0b00001f8b2
  7. Save the list on the Linux InfiniBand host as the /etc/opensm/guid.txt file.
  8. Set the permissions for the /etc/opensm/guid.txt file:
    # chmod 0644 /etc/opensm/guid.txt
    #
  9. Restart the Subnet Manager on the Linux InfiniBand host and instruct it to use the guid.txt file.

    For example:

    # /etc/init.d/opensmd stop
    Stopping IB Subnet Manager..-.                             [  OK  ]
    # /usr/sbin/opensm -R ftree -A -a /etc/opensm/guid.txt -s 120 -B
  10. Power on the fabric cards.

    See Power On the Fabric Cards.

Related Information