Exit Print View

Sun Datacenter InfiniBand Switch 648 Topic Set

Get PDF Book Print View
 

Document Information

Using This Documentation

Related Documentation

Documentation, Support, and Training

Documentation Feedback

Installation

Preparing the Site

Installing the Switch

Understanding the Installation

Supported Configurations

Fabric Card Distribution

Fabric Card Handling Guidelines

Line Card Distribution

Line Card Handling Guidelines

Fabric Card and Line Card Fillers

Installation Preparation

Suggested Tools for Installation

Antistatic Precautions for Installation

Installation Responsibilities

Installation Sequence

Installing the InfiniBand Software Stack

Install the HPC Software Suite

Unpacking the Chassis

Remove the Shipping Carton Contents

Unsecure the Chassis From the Pallet

Preparing the Chassis

Install the Rail Assemblies in the Rack

Install the Cable Trees

Install the Cable Curves

Install the Chassis in the Rack

Inspecting the Components

Inspecting the Midplane for Installation

Identify the Midplane Fabric Card Connectors

Inspect the Pins on the Rear Side of the Midplane

Identify the Midplane Line Card Connectors

Inspect the Pins on the Front Side of the Midplane

Inspecting the Fabric Cards for Installation

Identify the Fabric Cards for Installation

Inspect the Fabric Card Chassis for Installation

Inspect the Fabric Card Retainer Bolts for Installation

Inspect the Fans for Installation

Inspect the Fabric Card XBOW Connectors for Installation

Inspecting the Line Cards for Installation

Identify the Line Cards for Installation

Inspect the Line Card Chassis for Installation

Inspect the Line Card Retainer Bolts for Installation

Inspect the Line Card XBOW Connectors for Installation

Inspect the Line Card CXP Connectors for Installation

Inspecting the InfiniBand Cables for Installation

Identify the InfiniBand Cable for Installation

Inspect the InfiniBand Cable Hardware for Installation

Inspect the InfiniBand Cable Connectors for Installation

Installing the Components

Install the Midplane Stiffener

Install the Fabric Cards or Fabric Card Fillers

Remove the Midplane Stiffener

Install the Line Cards or Line Card Fillers

Powering On the CMCs

Attach the Management Cables

Attach the Power Cords

Check the Power Supply Status LEDs

Check the CMC Status LEDs

Accessing the CMC

Access the CMC From the Serial Management Port

Set the Initial UBoot Parameters

Access the CMC From the Network Management Port

Verify the CMC Status

Identify the CMC Addresses

Display the CMC FRU ID Information

Verify the Power Supply Status

Identify the Power Supply Addresses

Display the Power Supply FRU ID Information

Powering On the Fabric Cards

Enable the Fabric Cards for Installation

Create the guid.txt File for Installation

Power On the Fabric Cards for Installation

Check the Fabric Card Status LEDs

Verify the Fabric Card Status

Identify the Fabric Card Addresses

Display the Fabric Card FRU ID Information

Powering On the Line Cards

Enable the Line Cards for Installation

Power On the Line Cards for Installation

Check the Line Card Status LEDs

Verify the Line Card Status

Identify the Line Card Addresses

Display the Line Card FRU ID Information

Connecting InfiniBand Cables

Attach Cables to the CXP Connectors

Check the CXP Connector Status LEDs

Check Link Status

Identify the CXP Connector Addresses

Display the InfiniBand Cable FRU ID Information

Verifying the InfiniBand Fabric

Discover the InfiniBand Fabric Topology

Perform Diagnostics on the InfiniBand Fabric

Validate the InfiniBand Fabric and Report Errors

Administration

Administering the Switch

Diagnosing a Problem

Remote Management

Understanding ILOM on the Switch

Installing the ILOM Firmware

Administering ILOM (CLI)

Administering ILOM (Web Interface)

Administering ILOM (SNMP)

Administering Hardware (IPMI)

Understanding ILOM Commands

Service

Servicing the Switch

Reference

Command Reference

Index

Create the guid.txt File for Installation

If your InfiniBand fabric uses the Fat-Tree routing algorithm, you must provide a modified listing of fabric card GUIDs, called the root node GUID file, for the Subnet Manager. The /etc/opensm/guid.txt file is such a file. This procedure describes how to create the guid.txt file.


Note - In this procedure, you issue commands to both the CMC and the Linux InfiniBand host.


  1. Identify the prerequisite and subsequent tasks you must perform in conjunction with this procedure, to install the switch.

    See the table in Installation Sequence.

  2. Type the following command on the CMC to retrieve the fabric card base GUID:

    # getbaseguid fc slot

    where slot is the number of the fabric card (08). For example:

    # getbaseguid fc 0
    Base GUID = 0x00bad0fc0009f000
    #
  3. Record the GUID to a text editor on your Linux InfiniBand host.

  4. Repeat Step 2 and Step 3 for each fabric card.

    Once you have recorded each fabric card’s GUID, you will have a list similar to the following on your Linux InfiniBand host:

    0x00bad0fc0009f000
    0x00bad0fc0033f100
    0x00bad00017fcf200
    0x00bad00005fcf300
    0x00bad00003fcf400
    0x00bad00007fcf500
    0x00bad0fc0011f600
    0x00bad0b00002f700
    0x00bad0b00001f800

    Note - If you have fewer than 9 fabric cards, your list will be smaller.


  5. For each GUID, add the following values to make two new sums:

    • 0xa2

    • 0xb2


    Note - For GUIDs that end with two zeros (00), you are effectively replacing the zeros with a2 and b2.


    For example, 0x00bad0fc0009f000 becomes:

    • 0x00bad0fc0009f0a2

    • 0x00bad0fc0009f0b2

  6. Repopulate your list with the modified GUIDs.

    For example:

    0x00bad0fc0009f0a2
    0x00bad0fc0009f0b2
    0x00bad0fc0033f1a2
    0x00bad0fc0033f1b2
    0x00bad00017fcf2a2
    0x00bad00017fcf2b2
    0x00bad00005fcf3a2
    0x00bad00005fcf3b2
    0x00bad00003fcf4a2
    0x00bad00003fcf4b2
    0x00bad00007fcf5a2
    0x00bad00007fcf5b2
    0x00bad0fc0011f6a2
    0x00bad0fc0011f6b2
    0x00bad0b00002f7a2
    0x00bad0b00002f7b2
    0x00bad0b00001f8a2
    0x00bad0b00001f8b2
  7. Save the list on the Linux InfiniBand host as the /etc/opensm/guid.txt file.

  8. Set the permissions for the /etc/opensm/guid.txt file:

    # chmod 0644 /etc/opensm/guid.txt
    #
  9. Restart the Subnet Manager on the Linux InfiniBand host and instruct it to use the guid.txt file.

    For example:

    # /etc/init.d/opensmd stop
    Stopping IB Subnet Manager..-.                             [  OK  ]
    # /usr/sbin/opensm -R ftree -A -a /etc/opensm/guid.txt -s 120 -B
  10. Power on the fabric cards.

    See Power On the Fabric Cards for Installation.

Related Information