Sun Blade 6048 InfiniBand
QDR Switched Network Express Module User’s Guide | 
   |
| A P P E N D I X B |
This section provide some commands and typical outputs used to verify an InfiniBand (IB) network and the presence of each component in a Sun Blade 6048 Series Modular System shelf.
The IB-QNEM is installed in a Sun Blade 6048 Series Modular System shelf. The node or blade server has the following software installed:
Use any of the three commands in the example to display the local Host’s IB device status.
# ibstat
CA ’mlx4_0’
CA type: MT26428
Number of ports: 1
Firmware version: 2.6.0
Hardware version: a0
Node GUID: 0x50800200008e4d38
System image GUID: 0x50800200008e4d3b
Port 1:
State: Active
Physical state: LinkUp
Base lid: 7
Rate: 40
LMC: 0
SM lid: 13
Capability mask: 0x02510868
Port GUID: 0x50800200008e4d39
|
# ibv_devinfo
hca_id: mlx4_0
fw_ver: 2.6.000
node_guid: 5080:0200:008e:4d38
sys_image_guid: 5080:0200:008e:4d3b
vendor_id: 0x02c9
vendor_part_id: 26428
hw_ver: 0xA0
board_id: X6275_QDR_IB_1.8
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 13
port_lid: 7
port_lmc: 0x00
|
# ibstatus
Infiniband device ’mlx4_0’ port 1 status:
default gid: fe80:0000:0000:0000:5080:0200:008e:4d39
base lid: 0x7
sm lid: 0xd
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
|
Use the ibswitch command to verify that an IB-QNEM is installed in the shelf.
# ibswitch Switch : 0x002128183ea40050 ports 36 "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" base port 0 lid 4 lmc 0CA ’mlx4_0’ Switch : 0x002128183ea40040 ports 36 "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" base port 0 lid 3 lmc 0 CA type: MT26428 |
Use the ibhosts command to list all hosts in the IB network.
# ibhosts Ca : 0x50800200008e4974 ports 1 "ib-110 HCA-1" Ca : 0x50800200008e4970 ports 1 "ib-111 HCA-1" Ca : 0x50800200008e4954 ports 1 "ib-121 HCA-1" Ca : 0x50800200008e4950 ports 1 "ib-120 HCA-1" Ca : 0x50800200008e4ccc ports 1 "ib-107 HCA-1" Ca : 0x50800200008e48f4 ports 1 "ib-115 HCA-1" Ca : 0x50800200008e48f0 ports 1 "ib-114 HCA-1" Ca : 0x50800200008e4934 ports 1 "ib-119 HCA-1" Ca : 0x50800200008e4930 ports 1 "ib-118 HCA-1" Ca : 0x50800200008e4924 ports 1 "ib-123 HCA-1" Ca : 0x50800200008e4920 ports 1 "ib-122 HCA-1" Ca : 0x50800200008e4a7c ports 1 "ib-117 HCA-1" Ca : 0x50800200008e4a78 ports 1 "ib-116 HCA-1" Ca : 0x50800200008e4964 ports 1 "ib-108 HCA-1" Ca : 0x50800200008e4960 ports 1 "ib-109 HCA-1" Ca : 0x50800200008e5424 ports 1 "ib-101 HCA-1" Ca : 0x50800200008e5420 ports 1 "ib-100 HCA-1" Ca : 0x50800200008e4914 ports 1 "ib-105 HCA-1" Ca : 0x50800200008e4910 ports 1 "ib-104 HCA-1" Ca : 0x50800200008e4d3c ports 1 "ib-103 HCA-1" Ca : 0x50800200008e48e4 ports 1 "ib-113 HCA-1" Ca : 0x50800200008e48e0 ports 1 "ib-112 HCA-1" Ca : 0x50800200008e4d38 ports 1 "ib-102 HCA-1 |
Use the iblinkinfo command to list the network topology for the shelf.
# iblinkinfo.pl -R Switch 0x002128183ea40040 Sun Blade 6048 InfiniBand QDR Switched NEM I4A: InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 1[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 1[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 2[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 2[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 3[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 3[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 4[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 4[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 5[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 5[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 6[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 6[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 7[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 7[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 8[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 8[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 9[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 9[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 10[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 10[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 11[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 11[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 12[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 12[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 13[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 13[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 14[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 14[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 15[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 15[ ] "Sun Blade 6048 ( Could be 5.0 Gbps) 3 16[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 13 1[ ] "ib-122 HCA-1" ( Could be 5.0 Gbps) 3 17[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 14 1[ ] "ib-123 HCA-1" ( Could be 5.0 Gbps) 3 18[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 5 1[ ] "ib-118 HCA-1" ( Could be 5.0 Gbps) 3 19[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 8 1[ ] "ib-119 HCA-1" ( Could be 5.0 Gbps) 3 20[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 16 1[ ] "ib-114 HCA-1" ( Could be 5.0 Gbps) 3 21[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 10 1[ ] "ib-115 HCA-1" ( Could be 5.0 Gbps) 3 22[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 67 1[ ] "ib-222 HCA-1" ( Could be 5.0 Gbps) 3 23[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 98 1[ ] "ib-107 HCA-1" ( Could be 5.0 Gbps) 3 24[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 1 1[ ] "ib-120 HCA-1" ( Could be 5.0 Gbps) 3 25[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 15 1[ ] "ib-121 HCA-1" ( Could be 5.0 Gbps) 3 26[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 6 1[ ] "ib-111 HCA-1" ( Could be 5.0 Gbps) 3 27[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 19 1[ ] "ib-110 HCA-1" InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 28[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 28[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 29[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 29[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 30[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 30[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 31[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 31[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 32[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 32[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 33[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 33[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 34[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 34[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 35[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 35[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 36[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 36[ ] "Sun Blade 6048 I4B:Switch 0x002128183ea40050 Sun Blade 6048 InfiniBand QDR Switched NEM InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 1[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 1[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 2[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 2[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 3[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 3[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 4[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 4[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 5[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 5[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 6[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 6[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 7[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 7[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 8[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 8[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 9[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 9[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 10[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 10[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 11[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 11[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 12[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 12[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 13[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 13[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 14[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 14[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 15[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 15[ ] "Sun Blade 6048 ( Could be 5.0 Gbps) 4 16[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 20 1[ ] "ib-112 HCA-1" ( Could be 5.0 Gbps) 4 17[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 21 1[ ] "ib-113 HCA-1" ( Could be 5.0 Gbps) 4 18[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 7 1[ ] "ib-102 HCA-1" ( Could be 5.0 Gbps) 4 19[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 84 1[ ] "ib-103 HCA-1" ( Could be 5.0 Gbps) 4 20[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 24 1[ ] "ib-104 HCA-1" ( Could be 5.0 Gbps) 4 21[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 28 1[ ] "ib-105 HCA-1" ( Could be 5.0 Gbps) 4 22[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 9 1[ ] "ib-100 HCA-1" ( Could be 5.0 Gbps) 4 23[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 82 1[ ] "ib-101 HCA-1" ( Could be 5.0 Gbps) 4 24[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 17 1[ ] "ib-109 HCA-1" ( Could be 5.0 Gbps) 4 25[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 18 1[ ] "ib-108 HCA-1" ( Could be 5.0 Gbps) 4 26[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 2 1[ ] "ib-116 HCA-1" ( Could be 5.0 Gbps) 4 27[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 22 1[ ] "ib-117 HCA-1" InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 28[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 28[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 29[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 29[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 30[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 30[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 31[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 31[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 32[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 32[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 33[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 33[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 34[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 34[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 35[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 35[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 36[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 36[ ] "Sun Blade 6048 |
Use the mst and flint commands display the host’s firmware information.
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST Calibre (I2C) module is not loaded
MST devices:
------------
/dev/mst/mt26428_pciconf0 - PCI configuration cycles access.
bus:dev.fn=07:00.0 addr.reg=88 data.reg=92
Chip revision is: A0
/dev/mst/mt26428_pci_cr0 - PCI direct access.
bus:dev.fn=07:00.0 bar=0xfaf00000 size=0x100000
Chip revision is: A0
/dev/mst/mt26428_pci_msix0 - PCI direct access.
bus:dev.fn=07:00.0 bar=0x00000000 size=0x0
/dev/mst/mt26428_pci_uar0 - PCI direct access.
bus:dev.fn=07:00.0 bar=0xf3800000 size=0x800000
#
|
# flint -d /dev/mst/mt26428_pci_cr0 q Image type: ConnectX FW Version: 2.6.0 Rom Info: type=GPXE version=1.9.972 devid=26428 Device ID: 26428 Chip Revision: A0 Description: Node Port1 Port2 Sys image 50800200008e4d3bGUIDs: 50800200008e4d38 50800200008e4d39 50800200008e4d3a MACs: 00144fca3806 00144fca3807 Board ID: n/a (X6275_QDR_IB_1.8) VSD: n/a PSID: X6275_QDR_IB_1.8 |
Use the ifconfig and ping commands in the example to verify IPoIB.
# ifconfig
ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:100.1.1.102 Bcast:100.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::5280:200:8e:4d39/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:51 errors:0 dropped:9 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 b) TX bytes:8823 (8.6 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:560 (560.0 b) TX bytes:560 (560.0 b)
# ping 100.1.1.100
PING 100.1.1.100 (100.1.1.100) 56(84) bytes of data.
64 bytes from 100.1.1.100: icmp_seq=1 ttl=64 time=2.64 ms
64 bytes from 100.1.1.100: icmp_seq=2 ttl=64 time=0.093 ms
64 bytes from 100.1.1.100: icmp_seq=3 ttl=64 time=0.105 ms
--- 100.1.1.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.093/0.948/2.648/1.202 ms
[root@ib-102 ~]#
|
|
Sun Blade 6048 InfiniBand
QDR Switched Network Express Module User’s Guide | 820-6705-10 |
|
Copyright © 2009, Sun Microsystems, Inc. All rights reserved.
To Display the Local Host’s IB Device Status