A P P E N D I X B |
This section provide some commands and typical outputs used to verify an InfiniBand (IB) network and the presence of each component in a Sun Blade 6048 Series Modular System shelf.
The IB-QNEM is installed in a Sun Blade 6048 Series Modular System shelf. The node or blade server has the following software installed:
Use any of the three commands in the example to display the local Host’s IB device status.
# ibstat CA ’mlx4_0’ CA type: MT26428 Number of ports: 1 Firmware version: 2.6.0 Hardware version: a0 Node GUID: 0x50800200008e4d38 System image GUID: 0x50800200008e4d3b Port 1: State: Active Physical state: LinkUp Base lid: 7 Rate: 40 LMC: 0 SM lid: 13 Capability mask: 0x02510868 Port GUID: 0x50800200008e4d39 |
# ibv_devinfo hca_id: mlx4_0 fw_ver: 2.6.000 node_guid: 5080:0200:008e:4d38 sys_image_guid: 5080:0200:008e:4d3b vendor_id: 0x02c9 vendor_part_id: 26428 hw_ver: 0xA0 board_id: X6275_QDR_IB_1.8 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 13 port_lid: 7 port_lmc: 0x00 |
# ibstatus Infiniband device ’mlx4_0’ port 1 status: default gid: fe80:0000:0000:0000:5080:0200:008e:4d39 base lid: 0x7 sm lid: 0xd state: 4: ACTIVE phys state: 5: LinkUp rate: 40 Gb/sec (4X QDR) |
Use the ibswitch command to verify that an IB-QNEM is installed in the shelf.
# ibswitch Switch : 0x002128183ea40050 ports 36 "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" base port 0 lid 4 lmc 0CA ’mlx4_0’ Switch : 0x002128183ea40040 ports 36 "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" base port 0 lid 3 lmc 0 CA type: MT26428 |
Use the ibhosts command to list all hosts in the IB network.
# ibhosts Ca : 0x50800200008e4974 ports 1 "ib-110 HCA-1" Ca : 0x50800200008e4970 ports 1 "ib-111 HCA-1" Ca : 0x50800200008e4954 ports 1 "ib-121 HCA-1" Ca : 0x50800200008e4950 ports 1 "ib-120 HCA-1" Ca : 0x50800200008e4ccc ports 1 "ib-107 HCA-1" Ca : 0x50800200008e48f4 ports 1 "ib-115 HCA-1" Ca : 0x50800200008e48f0 ports 1 "ib-114 HCA-1" Ca : 0x50800200008e4934 ports 1 "ib-119 HCA-1" Ca : 0x50800200008e4930 ports 1 "ib-118 HCA-1" Ca : 0x50800200008e4924 ports 1 "ib-123 HCA-1" Ca : 0x50800200008e4920 ports 1 "ib-122 HCA-1" Ca : 0x50800200008e4a7c ports 1 "ib-117 HCA-1" Ca : 0x50800200008e4a78 ports 1 "ib-116 HCA-1" Ca : 0x50800200008e4964 ports 1 "ib-108 HCA-1" Ca : 0x50800200008e4960 ports 1 "ib-109 HCA-1" Ca : 0x50800200008e5424 ports 1 "ib-101 HCA-1" Ca : 0x50800200008e5420 ports 1 "ib-100 HCA-1" Ca : 0x50800200008e4914 ports 1 "ib-105 HCA-1" Ca : 0x50800200008e4910 ports 1 "ib-104 HCA-1" Ca : 0x50800200008e4d3c ports 1 "ib-103 HCA-1" Ca : 0x50800200008e48e4 ports 1 "ib-113 HCA-1" Ca : 0x50800200008e48e0 ports 1 "ib-112 HCA-1" Ca : 0x50800200008e4d38 ports 1 "ib-102 HCA-1 |
Use the iblinkinfo command to list the network topology for the shelf.
# iblinkinfo.pl -R Switch 0x002128183ea40040 Sun Blade 6048 InfiniBand QDR Switched NEM I4A: InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 1[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 1[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 2[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 2[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 3[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 3[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 4[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 4[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 5[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 5[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 6[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 6[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 7[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 7[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 8[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 8[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 9[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 9[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 10[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 10[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 11[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 11[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 12[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 12[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 13[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 13[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 14[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 14[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 15[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 15[ ] "Sun Blade 6048 ( Could be 5.0 Gbps) 3 16[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 13 1[ ] "ib-122 HCA-1" ( Could be 5.0 Gbps) 3 17[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 14 1[ ] "ib-123 HCA-1" ( Could be 5.0 Gbps) 3 18[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 5 1[ ] "ib-118 HCA-1" ( Could be 5.0 Gbps) 3 19[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 8 1[ ] "ib-119 HCA-1" ( Could be 5.0 Gbps) 3 20[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 16 1[ ] "ib-114 HCA-1" ( Could be 5.0 Gbps) 3 21[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 10 1[ ] "ib-115 HCA-1" ( Could be 5.0 Gbps) 3 22[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 67 1[ ] "ib-222 HCA-1" ( Could be 5.0 Gbps) 3 23[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 98 1[ ] "ib-107 HCA-1" ( Could be 5.0 Gbps) 3 24[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 1 1[ ] "ib-120 HCA-1" ( Could be 5.0 Gbps) 3 25[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 15 1[ ] "ib-121 HCA-1" ( Could be 5.0 Gbps) 3 26[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 6 1[ ] "ib-111 HCA-1" ( Could be 5.0 Gbps) 3 27[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 19 1[ ] "ib-110 HCA-1" InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 28[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 28[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 29[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 29[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 30[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 30[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 31[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 31[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 32[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 32[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 33[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 33[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 34[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 34[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 35[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 35[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4B" ( Could be 5.0 Gbps) 3 36[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 4 36[ ] "Sun Blade 6048 I4B:Switch 0x002128183ea40050 Sun Blade 6048 InfiniBand QDR Switched NEM InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 1[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 1[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 2[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 2[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 3[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 3[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 4[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 4[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 5[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 5[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 6[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 6[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 7[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 7[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 8[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 8[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 9[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 9[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 10[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 10[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 11[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 11[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 12[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 12[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 13[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 13[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 14[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 14[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 15[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 15[ ] "Sun Blade 6048 ( Could be 5.0 Gbps) 4 16[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 20 1[ ] "ib-112 HCA-1" ( Could be 5.0 Gbps) 4 17[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 21 1[ ] "ib-113 HCA-1" ( Could be 5.0 Gbps) 4 18[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 7 1[ ] "ib-102 HCA-1" ( Could be 5.0 Gbps) 4 19[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 84 1[ ] "ib-103 HCA-1" ( Could be 5.0 Gbps) 4 20[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 24 1[ ] "ib-104 HCA-1" ( Could be 5.0 Gbps) 4 21[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 28 1[ ] "ib-105 HCA-1" ( Could be 5.0 Gbps) 4 22[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 9 1[ ] "ib-100 HCA-1" ( Could be 5.0 Gbps) 4 23[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 82 1[ ] "ib-101 HCA-1" ( Could be 5.0 Gbps) 4 24[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 17 1[ ] "ib-109 HCA-1" ( Could be 5.0 Gbps) 4 25[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 18 1[ ] "ib-108 HCA-1" ( Could be 5.0 Gbps) 4 26[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 2 1[ ] "ib-116 HCA-1" ( Could be 5.0 Gbps) 4 27[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 22 1[ ] "ib-117 HCA-1" InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 28[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 28[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 29[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 29[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 30[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 30[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 31[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 31[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 32[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 32[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 33[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 33[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 34[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 34[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 35[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 35[ ] "Sun Blade 6048 InfiniBand QDR Switched NEM I4A" ( Could be 5.0 Gbps) 4 36[ ] ==( 4X 10.0 Gbps Active / LinkUp)==> 3 36[ ] "Sun Blade 6048 |
Use the mst and flint commands display the host’s firmware information.
# mst status MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST Calibre (I2C) module is not loaded MST devices: ------------ /dev/mst/mt26428_pciconf0 - PCI configuration cycles access. bus:dev.fn=07:00.0 addr.reg=88 data.reg=92 Chip revision is: A0 /dev/mst/mt26428_pci_cr0 - PCI direct access. bus:dev.fn=07:00.0 bar=0xfaf00000 size=0x100000 Chip revision is: A0 /dev/mst/mt26428_pci_msix0 - PCI direct access. bus:dev.fn=07:00.0 bar=0x00000000 size=0x0 /dev/mst/mt26428_pci_uar0 - PCI direct access. bus:dev.fn=07:00.0 bar=0xf3800000 size=0x800000 # |
# flint -d /dev/mst/mt26428_pci_cr0 q Image type: ConnectX FW Version: 2.6.0 Rom Info: type=GPXE version=1.9.972 devid=26428 Device ID: 26428 Chip Revision: A0 Description: Node Port1 Port2 Sys image 50800200008e4d3bGUIDs: 50800200008e4d38 50800200008e4d39 50800200008e4d3a MACs: 00144fca3806 00144fca3807 Board ID: n/a (X6275_QDR_IB_1.8) VSD: n/a PSID: X6275_QDR_IB_1.8 |
Use the ifconfig and ping commands in the example to verify IPoIB.
# ifconfig ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:100.1.1.102 Bcast:100.1.1.255 Mask:255.255.255.0 inet6 addr: fe80::5280:200:8e:4d39/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:51 errors:0 dropped:9 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:0 (0.0 b) TX bytes:8823 (8.6 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:560 (560.0 b) TX bytes:560 (560.0 b) # ping 100.1.1.100 PING 100.1.1.100 (100.1.1.100) 56(84) bytes of data. 64 bytes from 100.1.1.100: icmp_seq=1 ttl=64 time=2.64 ms 64 bytes from 100.1.1.100: icmp_seq=2 ttl=64 time=0.093 ms 64 bytes from 100.1.1.100: icmp_seq=3 ttl=64 time=0.105 ms --- 100.1.1.100 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.093/0.948/2.648/1.202 ms [root@ib-102 ~]# |
Copyright © 2009, Sun Microsystems, Inc. All rights reserved.