JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Sun Network QDR InfiniBand Gateway Switch HTML Document Collection for Firmware Version 2.0
search filter icon
search icon

Document Information

Using This Documentation

Product Notes

Related Documentation

Feedback

Support and Accessibility

Installation

Understanding the Gateway

Gateway Overview

Physical Specifications

Environmental Requirements

Acoustic Noise Emissions

Electrical Specifications

Network Management Connector and Pins

USB Management Connector and Pins

Data QSFP Connector and Pins

Understanding Cabling

Routing Service Cables

Understanding Data Cabling

Preparing to Install the Gateway

Installation Preparation

Suggested Tools for Installation

Antistatic Precautions for Installation

Installation Responsibilities

Installation Sequence

Verify Shipping Carton Contents

Assemble the Ethernet Splitter Data Cables

Route the Data Cables

Installing the Gateway

Install the Gateway in the Rack

Powering On the Gateway

Connecting Data cables

Verifying the InfiniBand Fabric

Administration

Troubleshooting the Gateway

Gateway Hardware Problems

InfiniBand Fabric Problems

Identifying LEDs

Understanding Routing Through the Gateway

Switch GUIDs Overview

Understanding Administrative Commands

Configuration Overview

Hardware Command Overview

InfiniBand Command Overview

ILOM Command Overview

Administering the Chassis

Monitoring the Chassis

Controlling the Chassis

Administering the I4 Switch Chip

Monitoring the I4 Switch Chip

Controlling the I4 Switch Chip

Administering the Subnet Manager

Monitoring the Subnet Manager

Controlling the Subnet Manager

Administering the InfiniBand Fabric

Monitoring the InfiniBand Fabric

Controlling the InfiniBand Fabric

Configuring the Fabric Director Node List

Partitioning the InfiniBand Fabric

Administering Gateway Resources

Installing Gateway Supportive Software (Linux)

Acquire the BXOFED Software (Linux)

Install the BXOFED Software (Linux)

Creating VNICs Under Gateway Manual Mode (Linux)

Gateway Manual Mode Overview (Linux)

Determine VNIC Configuration Parameters For Gateway Manual Mode (Linux)

Determine VLAN Associations for Gateway Manual Mode (Linux)

Configure and Create VNICs for Gateway Manual Mode (Linux)

Creating VNICs Under Host Manual Mode (Linux)

Host Manual Mode Overview (Linux)

Central Configuration File (Linux)

VNIC-Specific Configuration File (Linux)

Determine VNIC Configuration Parameters for Host Manual Mode (Linux)

Create the Central Confguration File (Linux)

Create the VNIC-Specific Confguration Files (Linux)

mlx4_vnic_confd Daemon (Linux)

Configure and Create VNICs for Host Manual Mode (Linux)

Creating Virtual IO Adapters (Oracle Solaris)

Ethernet Over InfiniBand Overview (Oracle Solaris)

Verify That the Ethernet Over InfiniBand Driver Is Installed (Oracle Solaris)

Install the Ethernet Over InfiniBand Driver (Oracle Solaris)

Set Up VIOAs (Oracle Solaris)

Configure IP Addresses on the Data Link (Oracle Solaris)

Create Oracle Solaris VNICs (Oracle Solaris)

Monitoring Gateway Resources

Display LAG Information

Display the VLANs

Display the VNICs

Display Gateway Ethernet Port Information

Display Gateway Port Information

Display the Gateway Instance Number

Display Information About the Gateway

Controlling LAGs

Enable LAG Mode

Create LAGs

Add or Delete Connectors From a LAG

Delete a LAG

Disable LAG Mode

Controlling VLANs and VNICs

Create VLANs

Delete VLANs

Create VNICs

Enable VNICs

Disable VNICs

Delete VNICs

Controlling Gateway Ports and Parameters

Enable a Gateway Ethernet Port

Disable a Gateway Ethernet Port

Set the Gateway Ethernet Port Parameters

Set the Gateway Service Level

Set the Gateway Instance Number

Set the Gateway System Name

Remote Management

Understanding Oracle ILOM on the Gateway

Oracle ILOM Overview

Supported Features

Understanding Oracle ILOM Targets

Administering Oracle ILOM (CLI)

CLI Overview

Accessing Oracle ILOM From the CLI

Switching Between the Oracle ILOM Shell and the Linux Shell

Monitoring Oracle ILOM Targets (CLI)

Controlling Oracle ILOM Targets (CLI)

Upgrading the Gateway Firmware Through Oracle ILOM (CLI)

Administering Oracle ILOM (Web)

Web Interface Overview

Access Oracle ILOM From the Web Interface

Monitoring Oracle ILOM Targets (Web)

Controlling Oracle ILOM Targets (Web)

Upgrade the Gateway Firmware (Web)

Using the Fabric Monitor

Access the Fabric Monitor

Fabric Monitor Features

Accessing the Rear Panel Diagram

Accessing Status Pane Information

Control Panel Function

Monitoring Parameters and Status

Administering Oracle ILOM (SNMP)

SNMP Overview

Understanding SNMP Commands

Monitoring Oracle ILOM Targets (SNMP)

Controlling Oracle ILOM Targets (SNMP)

Administering Hardware (IPMI)

ipmitool Overview

Display the Sensor States (IPMI)

Display the Sensor Information (IPMI)

Display the System Event Log (IPMI)

Display FRU ID Information (IPMI)

Display Gateway Status LED States (IPMI)

Enable the Locator LED (IPMI)

Disable the Locator LED (IPMI)

Understanding Oracle ILOM Commands

cd Command

create Command

delete Command

dump Command

exit Command (ILOM)

help Command (ILOM)

load Command

reset Command

set Command

show Command

version Command (ILOM)

Service

Understanding Service Procedures

Replaceable Components

Suggested Tools for Service

Antistatic Precautions for Service

Servicing Power Supplies

Inspecting a Power Supply

Power Off a Power Supply

Remove a Power Supply

Install a Power Supply

Power On a Power Supply

Servicing Fans

Inspecting a Fan

Remove a Fan

Install a Fan

Servicing Data Cables

Inspecting the Data Cables

Remove a Data Cable

Install a Data Cable

Servicing the Battery

Remove the Gateway From the Rack

Replace the Battery

Reference

Understanding Hardware Commands

Linux Shells for Hardware Commands

addlagport Command

allowhostconfig Command

checkboot Command

checkpower Command

checktopomax Command

checkvoltages Command

connector Command

createlag Command

createvlan Command

createvnic Command

dcsport Command

deletelag Command

deletevlan Command

deletevnic Command

dellagport Command

disablecablelog Command

disablegwport Command

disablelagmode Command

disablelinklog Command

disablesm Command

disableswitchport Command

disablevnic Command

disallowhostconfig Command

enablecablelog Command

enablegwport Command

enablelagmode Command

enablelinklog Command

enablesm Command

enableswitchport Command

enablevnic Command

env_test Command

exit Command (Hardware)

fdconfig Command

fwverify Command

generatetopology Command

getfanspeed Command

getmaster Command

getportcounters Command

getportstatus Command

help Command (Hardware)

listlinkup Command

matchtopology Command

setcontrolledhandover Command

setdefaultgwdiscpkey Command

setgwethport Command

setgwinstance Command

setgwsl Command

setgwsystemname Command

setmsmlocationmonitor Command

setsmmkey Command

setsmpriority Command

setsubnetprefix Command

showfruinfo Command

showgwconfig Command

showgwports Command

showlag Command

showpsufru Command

showsmlog Command

showtemps Command

showtopology Command

showunhealthy Command

showvlan Command

showvnics Command

smconfigtest Command

smnodes Command

smpartition Command

version Command (Hardware)

Understanding InfiniBand Commands

Linux Shells for InfiniBand Commands

ibdiagnet Command

ibhosts Command

ibnetdiscover Command

ibnetstatus Command

ibnodes Command

ibportstate Command

ibroute Command

ibrouters Command

ibstat Command

ibswitches Command

ibtracert Command

perfquery Command

saquery Command

smpquery Command

Understanding SNMP MIB OIDs

OID Tables Overview

Understanding the SUN-DCS-IB-MIB MIB OIDs

Understanding the SUN-FABRIC-MIB MIB OIDs

Understanding the SUN-HW-TRAP-MIB MIB OIDs

Understanding the SUN-ILOM-CONTROL-MIB MIB OIDs

Understanding the SUN-PLATFORM-MIB MIB OIDs

Understanding the ENTITY-MIB MIB OIDs

Index

Set Up VIOAs (Oracle Solaris)

  1. Become superuser of the Oracle Solaris 11 host.

    For this procedure, the Oracle Solaris 11 host name is solaris01.

  2. Display the HCA GUIDs and port GUIDs configured on the host.
    # dladm show-ib
    LINK         HCAGUID         PORTGUID        PORT STATE  PKEYS
    ibp0         21280001A0A590  21280001A0A591  1    up     FFFF
    ibp1         21280001A0A590  21280001A0A592  2    up     FFFF
    #
  3. Record the host name, HCA GUIDs, and port GUIDs.

    In this example, the HCA GUID is 21280001A0A590 and the port GUIDs are 21280001A0A591 and 21280001A0A592.

  4. From the management controller of the gateway, display the HCAs recognized by the gateway.
    # ibhosts
    Ca      : 0x0021280001A0A590 ports 2 "solaris01"
    Ca      : 0x00212856cd22c040 ports 2 "SUN IB QDR GW switch mnm34-97 Bridge 1"
    Ca      : 0x0002c903000891aa ports 2 "mnm34-54 HCA-1"
    Ca      : 0x00212800013ece9e ports 2 "mnm34-55 HCA-1"
    Ca      : 0x0003ba000100e370 ports 2 "mnm34-60 HCA-1"
    .
    .
    .
    #
  5. Compare the output of the ibhosts command with the recorded host name, HCA GUIDs, and port GUIDs.

    The port GUIDs are the HCA GUID +1 (port 1) and HCA GUID +2 (port 2), respectively.

  6. Verify that the gateway recognizes the Oracle Solaris 11 host correctly.

    If not, check the cabling between the gateway and the Oracle Solaris 11 host.

  7. Determine if a VIOA has already been assigned to the Oracle Solaris 11 host.
    # showvnics
    ID  STATE     FLG IOA_GUID                NODE       IID  MAC               VLN PKEY   GW
    --- --------  --- ----------------------- ---------- ---- ----------------- --- ----   --------
     4 UP          N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:03 NO  ffff   0A-ETH-1
     5 UP          N 00:03:BA:00:01:00:E3:71 mnm34-60 0002 02:02:02:02:02:04 NO  ffff   0A-ETH-1
     3 DISABLED    N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:02 NO  ffff   0A-ETH-1
     6 UP          N 00:03:BA:00:01:00:E3:72 mnm34-60 0000 02:02:02:02:02:05 NO  ffff   0A-ETH-1
    #
  8. Examine the output by column:
    • STATE – The state of the VIOA.

    • IOA_GUID – The port GUID of the VIOA. The GUID might be listed several times.

    • NODE – The host name of the node hosting the VIOA.

    • MAC – The MAC address assigned to the VIOA. There might be several MAC addresses.

    • PKEY – The partition key assigned.

    • GW – The physical connector cabled to the node or host.

  9. Determine your next steps.
  10. Determine your MAC addresses and whether they are to be global or locally administrated.

    Note - Each MAC address must be unique and not 00:00:00:00:00:00.



    Note - Only even numbers are supported for the most significant byte of the MAC address (unicast).


  11. Activate the VIOA by assigning a MAC address to it.
    # createvnic connector -guid guid -mac mac_address -pkey default

    where:

    • connector is the name of the connector (0A-ETH-1 to 0A-ETH-4 and 1A-ETH-1 to 1A-ETH-4).

    • guid is the global unique identifier of the target port on the host associated with the VIOA.

    • mac_address is the MAC address to be assigned to the VIOA.


    Note - You can use the information from Step 8 as a guide.


    For example, to assign the MAC address of a0:a5:91:95:30:9a to the VIOA for port GUID 21280001A0A591 through connector 0A-ETH-2:

    # createvnic 0A-ETH-2 -guid 0021280001A0A591 -mac a0:a5:91:95:30:9a -pkey default
    VNIC created
    #

    The VIOA is created and an Ethernet over InfiniBand data link is bound to the VIOA.

  12. Verify that the VIOA is active.
    # showvnics
    ID  STATE     FLG IOA_GUID                NODE       IID  MAC               VLN PKEY   GW
    --- --------  --- ----------------------- ---------- ---- ----------------- --- ----   --------
     7 UP          N 00:21:28:00:01:A0:A5:91 solaris01 0000 a0:a5:91:95:30:9a NO  ffff 0A-ETH-2
     4 UP          N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:03 NO  ffff   0A-ETH-1
     5 UP          N 00:03:BA:00:01:00:E3:71 mnm34-60 0002 02:02:02:02:02:04 NO  ffff   0A-ETH-1
     3 DISABLED    N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:02 NO  ffff   0A-ETH-1
     6 UP          N 00:03:BA:00:01:00:E3:72 mnm34-60 0000 02:02:02:02:02:05 NO  ffff   0A-ETH-1
    #
  13. Configure an IP address on the data link.

    See Configure IP Addresses on the Data Link (Oracle Solaris).

Related Information