Verifying the InfiniBand Network Fabric Network
This procedure describes how to verify the InfiniBand Network Fabric network.
-
Visually check all the RDMA Network Fabric cable connections within the rack. The port lights should be on, and the LEDs should be on. Do not press each connector to verify connectivity.
-
Log in as the
root
user on any component in the rack. -
Verify the InfiniBand Network Fabric topology using the following commands:
# cd /opt/oracle.SupportTools/ibdiagtools # ./verify-topology [-t rack_size]
The following example shows the output when the network components are correct.
[DB Machine Infiniband Cabling Topology Verification Tool ] Is every external switch connected to every internal switch......[SUCCESS ] Are any external switches connected to each other................[SUCCESS ] Are any hosts connected to spine switch..........................[SUCCESS ] Check if all hosts have 2 CAs to different switches..............[SUCCESS ] Leaf switch check:cardinality and even distribution..............[SUCCESS ] Check if each rack has an valid internal ring....................[SUCCESS ]
In the preceding command, rack_size is the size of the rack. The
-t rack_size
option is needed if the rack is Recovery Appliance Half Rack or Recovery Appliance Quarter Rack. Use the valuehalfrack
orquarterrack
, if needed.The following example shows the output when there is a bad RDMA Network Fabric switch to cable connection:
#./verify-topology [DB Machine Infiniband Cabling Topology Verification Tool ] Is every external switch connected to every internal switch......[SUCCESS ] Are any external switches connected to each other................[SUCCESS ] Are any hosts connected to spine switch..........................[SUCCESS ] Check if all hosts have 2 CAs to different switches..............[SUCCESS ] Leaf switch check:cardinality and even distribution..............[SUCCESS ] Check if each rack has an valid internal ring....................[ERROR ] Switches 0x21283a87cba0a0 0x21283a87b8a0a0 have 6 connections between them. They should have at least 7 links between them
The following example shows the output when there is a bad RDMA Network Fabric cable on a database server:
#./verify-topology [DB Machine Infiniband Cabling Topology Verification Tool ] Is every external switch connected to every internal switch......[SUCCESS ] Are any external switches connected to each other................[SUCCESS ] Are any hosts connected to spine switch..........................[SUCCESS ] Check if all hosts have 2 CAs to different switches..............[ERROR ] Node db01 has 1 endpoints.(Should be 2) Port 2 of this node is not connected to any switch --------fattree End Point Cabling verification failed----- Leaf switch check:cardinality and even distribution..............[ERROR ] Internal QDR Switch 0x21283a87b8a0a0 has fewer than 4 compute nodes It has only 3 links belonging to compute nodes [SUCCESS ] Check if each rack has an valid internal ring...................[SUCCESS ]
The following example shows the output when there is a bad connection on the switch and the system:
#./verify-topology [DB Machine Infiniband Cabling Topology Verification Tool ] Is every external switch connected to every internal switch......[SUCCESS ] Are any external switches connected to each other................[SUCCESS ] Are any hosts connected to spine switch..........................[SUCCESS ] Check if all hosts have 2 CAs to different switches..............[ERROR ] Node burxdb01 has 1 endpoints.(Should be 2) Port 2 of this node is not connected to any switch --------fattree End Point Cabling verifation failed----- Leaf switch check:cardinality and even distribution..............[ERROR ] Internal QDR Switch 0x21283a87b8a0a0 has fewer than 4 compute nodes It has only 3 links belonging to compute nodes...................[SUCCESS ] Check if each rack has an valid internal ring....................[ERROR ] Switches 0x21283a87cba0a0 0x21283a87b8a0a0 have 6 connections between them They should have at least 7 links between them