Monitoring and Troubleshooting IB Devices

In the Oracle Solaris 11 release, new commands and utilities enable you to manage the IB fabric more effectively. These commands are included in the system/io/infiniband/open-fabrics package and the man pages are installed automatically when the open-fabrics package is installed. For example:

% man rping
Reformatting page.  Please Wait... done

librdmacm                                                RPING(1)

NAME
     rping - RDMA CM connection and RDMA ping-pong test.

SYNOPSIS
     rping -s [-v] [-V] [-d] [-P] [-a address] [-p port]
               [-C message_count] [-S message_size]
     rping -c [-v] [-V] [-d] -a address [-p port]
               [-C message_count] [-S message_size]
.
.
.

The following new commands and utilities provide the ability to list and query IB devices, diagnose and trouble shoot IB fabric issues, and measure IB performance.

Table 6-1 General IB Monitoring Commands

Command	Description
`ibv_asyncwatch`	Monitors InfiniBand asynchronous events
`ibv_devices or ibv_devinfo`	Lists InfiniBand devices or device information
`ibv_rc_pingpong`, `ibv_srq_pingpong`, or `ibv_ud_pingpong`	Tests node to node connectivity by using RC connection, SRQs, or UD connection
`mckey`	Tests RDMA CM multicast setup and simple data transfer
`rping`	Tests RDMA CM connection and attempts RDMA ping-pong
`ucmatose`	Tests RDMA CM connection and attempts simple ping-pong
`udaddy`	Tests RDMA CM datagram setup and attempts simple ping-pong

Table 6-2 General IB Performance Testing Commands

Command	Description
`rdma_bw or rdma_lat`	Tests RDMA write transactions for streaming bandwidth or latency
`ib_read_bw` or `ib_read_lat`	Tests RDMA read transactions for bandwidth or latency
`ib_send_bw` or `ib_send_lat`	Tests RDMA send transactions for bandwidth or latency
`ib_write_bw` or `ib_write_bw_postlist`	Tests RDMA write transactions for bandwidth that displays one I/O request at a time or post list bandwidth that displays a list of I/O requests
`ib_write_lat`	Tests RDMA write transactions for latency
`ib_clock_test`	Tests accuracy of system clock
`qperf`	Measures socket and RDMA performance

Table 6-3 RDS Monitoring and Testing Tools

Command	Description
`rds-info`	Displays RDS kernel module information
`rds-ping`	Determines if remote node over RDS is reachable
`rds-stress`	Sends message between processes over RDS sockets

Table 6-4 Fabric Diagnostic Tools

Command	Description
`ibdiagnet`	Performs diagnostic check of the entire fabric
`ibaddr`	Queries InfiniBand address or addresses
`ibnetdiscover`	Discovers remote InfiniBand topology
`ibping`	Validates connectivity between IB nodes
`ibportstate`	Queries physical port state and link speed of an IB port
`ibroute`	Displays InfiniBand switch forwarding tables
`ibstat` or `ibsysstat`	Query status of InfiniBand device or devices or the status of a system on an IB address
`ibtracert`	Traces an IB path
`perfquery` or `saquery`	Queries IB port counters or sIB subnet administration attributes
`sminfo`	Queries IB SMInfo attribute
`smpquery` or `smpdump`	Queries or dumps IB subnet management attributes
`ibcheckerrors` or `ibcheckerrs`	Validates IB port (or node) or IB subnet and reports errors
`ibchecknet`, `ibchecknode`, or `ibcheckport`	Validates IB subnet, node, or port and reports errors
`ibcheckportstate, ibcheckportwidth`, `ibcheckstate`, or `ibcheckwidth`	Validates IB port that are link up but not active, ports for 1x (2.0 Gbps) link width, ports in IB subnet that are link up but not active, or lx links in IB subnet
`ibclearcounters` or `ibclearerrors`	Clears port counters or error counters in IB subnet
`ibdatacounters` or `ibdatacounts`	Queries for data counters in IB subnet or IB port data counters
`ibdiscover.pl`	Annotates and compares IB topology
`ibhosts`	Displays IB host nodes in topology
`iblinkinfo.pl` or `iblinkinfo`	Displays link information for all links in the fabric
`ibnodes`	Displays IB nodes in topology
`ibprintca.pl`	Displays either the CA specified or the list of CAs from the `ibnetdiscover` output
`ibprintrt.pl`	Displays either only the router specified or a list of routers from the `ibnetdiscover` output
`ibprintswitch.pl`	Displays either the switch specified or a list of switches from the `ibnetdiscover` output
`ibqueryerrors.pl`	Queries and report non-zero IB port counters
`ibrouters`	Displays IB router nodes in topology
`ibstatus`	Queries basic status of IB devices
`ibswitches`	Displays IB switch nodes in topology
`ibswportwatch.pl`	Polls the counters on the specified switch or port and report rate of change information
`set_nodedesc.sh`	Sets or displays node description string for IB Host Controller Adapters (HCA)s
`dump2psl.pl`	Dumps PSL file based on `opensm` output file that is used for credit loop checking
`dump2slvl.pl`	Dumps SLVL file based on `opensm` output file that is used for credit loop checking
`ibis`	An extended TCL shell for IB management inband services

Skip Navigation Links
Exit Print View
	Oracle Solaris 11.1 Administration: Devices and File Systems Oracle Solaris 11.1 Information Library