Before you can start creating unique InfiniBand partitions, you must complete the following tasks:
Ensure that the InfiniBand switches in your Exalogic machine are installed with firmware versions 2.0.4 or above. This requirement is mandatory.
Before creating an InfiniBand partition, you must identify the port GUIDs of Exalogic compute nodes that will be added to the partition. In addition, you must identify the BridgeX ports of the gateway switches that are connected to those Exalogic compute nodes.
Identifying Port GUIDs on Compute Nodes
To identify the port GUIDs on an Exalogic compute node, run the following command on the command line:
# ibstat
This command displays output, as in the following example:
CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.9.1000
Hardware version: b0
Node GUID: 0x0021280001cef972
System image GUID: 0x0021280001cef975
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 36
LMC: 0
SM lid: 5
Capability mask: 0x02510868
Port GUID: 0x0021280001cef973
Link layer: IB
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 37
LMC: 0
SM lid: 5
Capability mask: 0x02510868
Port GUID: 0x0021280001cef974
Link layer: IB
In the above example, Port GUID values are highlighted in a rectangle for illustration purposes only. The actual command does not highlight Port GUID. You must see the command output and note down the values for both InfiniBand ports on each compute node.
Alternatively, you can run the following command to display only GUIDs:
# ibstat | grep 'Port GUID:'
To identify the BridgeX ports on the gateway switches that are connected to your compute nodes, run the following command at the command prompt on each gateway switch that your compute node is connected to:
# showgwports
This command displays the BridgeX ports. Note down the values in the INTERNAL PORTS section of the output, as in the following example:
INTERNAL PORTS: --------------- Device Port Portname PeerPort PortGUID LID IBState GWState -------------------------------------------------------------------- Bridge-0 1 Bridge-0-1 4 0x002128548062c001 0x0015 Active Up Bridge-0 2 Bridge-0-2 3 0x002128548062c002 0x000d Active Up Bridge-1 1 Bridge-1-1 2 0x002128548062c041 0x000f Active Up Bridge-1 2 Bridge-1-2 1 0x002128548062c042 0x0010 Active Up
Tip:
In an Exalogic machine full rack, compute nodes 1 to 15 (start from the bottom of the rack) connect their InfiniBand port 1 to gateway switch1 and their InfiniBand port2 to gateway switch 2. Similarly, compute nodes 16 to 30 are connected to gateway switches 3 and 4.
To identify all InfiniBand switches (Sun Network InfiniBand Gateway Switch or Sun Datacenter InfiniBand Switch 36) running master or standby instances of Subnet Manager (SM) on the fabric, run the following command on any of the InfiniBand switches:
# ibswitches
This command displays the GUID, name, LID, and LMC for each switch. The output of the command is a mapping of GUID to LID for switches in the fabric.
After identifying the InfiniBand switches and their IP addresses, you must log in to each of the switches and run the following command to identify the InfiniBand switch where the master Subnet Manager (SM) is running:
# getmaster
This command displays output, as shown in the following example:
Local SM enabled and running 20111122 08:45:02 Master SubnetManager on sm lid 11 sm guid 0x21283bad45c0a0 : SUN IB QDR GW switch el01gw04 10.10.10.10
After identifying the InfiniBand switch where master SM is running, log in to the ILOM shell for the InfiniBand switch as the ILOM administrator (ilom-admin). After logging in, run the show /SYS/Fabric_Mgmt command to log in to the restricted Linux shell. To view a list of available commands, you can run the help all command.