|C H A P T E R 10|
Infiniband Host Channel Adapter Test (ibhcatest)
ibhcatest comprises multiple iRISC CPU cores, two 4x Infiniband ports, and integrated SerDes components. In addition, the ibhcatest external associated components include FLASH ROM and DDR memory. ibhcatest provides high speed interconnect through PCI interface to external Infiniband fabric. Supported platforms include: two 1U and two 2U x86 AMD Opteron entry-level servers, Sun Fire V2XX, V4XX and E-series high-end servers.
ibhcatest exercises and verifies the proper operation of the Tavor chip and its associated components, such as DDR memory, flash PROM, and internal IB packet transmit/receive circuitry. The isolates single faults to the identifiable component(s).
ibhcatest supports three execution test modes in SunVTS Connection, Exclusive and Functional. In Connection mode, the test queries for the Tavor firmware and hardware revision, and running internal loopback.
The internal loopback test is run at least once depending on the amount of time each pass takes. In Functional mode all subtests are executed according to the options selected. In Exclusive mode all subtests are executed in sequence.
Tavor supports an internal loopback mechanism that is very similar to the actual operation. The main difference is that data does not go through the integrated SerDes and the 4x IB port circuitry. On the receiving side, data does not get verified by the CRC algorithm. Otherwise, all other components of Tavor that involved in transmitting and receiving data packets are being exercised by ibhcatest.
Tavor-based HCA uses a single, 256 MB DDR memory for data storage at run time. This data storage is shared by three interdependent clients, Tavor driver, firmware, and hardware. During driver initialization, predetermined data structures and data are laid out in the memory.
With no exclusive access from the driver side, subsequent writes to any memory location that contain real data can cause undesirable results like a system crash. Furthermore, the data allocation size is fixed, writing to the remaining free memory does not add any value in terms of finding faults.
The memory subtest is limited to read only operations which cover the entire DDR memory. The test does not check for data corruption and no mechanism for triggering bit errors through writing to memory. The test uncovers bus related problems.
The HCA supports internal loopback for packets transmitted between QPs that are assigned to the same HCA port. If a packet is being transmitted to a DLID that is equivalent to the Port LID with the LMC bits masked out or the packet DLID is a multicast LID, the packet goes on the loopback path. In this latter case, the packet also is transmitted to the fabric. When a packet is looped back, it must pass the SL2VL mapping. If the mapping yields 15 or a nonoperational VL, the packet is discarded. In the inbound direction, the ICRC and VCRC checks are blindly passed for looped back packets.
Note that internal loopback is supported only for packets that are transmitted and received on the same port. Packets that are transmitted on one port and received on another port are transmitted to the fabric. The fabric directs these packets to the destination port.
This subtest uses interfaces from the Tavor driver to perform loopback testing. Information such as data pattern for data packets, port number, CQ polling, retries between iteration, and the number of iterations for each ioctl call are passed to the driver. Once finished, status regarding the number passes completed is returned. If the number of passes does not match the number of iterations, a failure has occurred. This might happen when the number of retries is exhausted, and the last failing buffer in the retry series is returned as result. SunVTS then determines exactly what failed in the buffer and reports the failure. The options for this subtest are as follows:
This subtest comprises two test modes, Sequential and Random. The start and end address offsets are determined dynamically by obtaining them from the firmware. In Sequential mode, length and starting offset are instructed by the test option rdoffset and rdsz. Then the test sequentially reads data from each memory address until all memory locations are covered or the end address is reached. Each read is accomplished by an ioctl call to the driver.
The test returns pass or fail based on the completion status of each ioctl call. In Random mode, this subtest reads the number of rdsz times in a randomly generated address bound by the start and end address offset. The options for this subtest are as follows:
Caution - In the SunVTS environment, ibhcatest and nettest are mutually exclusive. nettest has higher priority if the IB port interface is plumbed up when SunVTS is invoked. These two tests cannot be run at the same time at the command line If both of these tests are invoked at the command line, ibhcatest exits gracefully if the IB port interface is plumbed up. The commands to bring down the IB daemon (ibd [IPoIB]) are as follows:
To reach the following dialog box, right-click on the test name in the System Map and select Test Parameter Options. If you do not see this test in the System Map, you might need to expand the collapsed groups, or your system might not include the device appropriate to this test. Refer to the SunVTS User's Guide.
/opt/SUNWvts/bin/ibhcatest [-scruvdtlxnf] [-p n] [-i n] [-w n] [-o [dev=text] [lb=Enabled|Disabled] [tlbport=1+2] [data=Pattern] [cq=Time] [loop=Number] [ddr=Enabled|Disabled] [rdoffset=Offset] [rdsz=Len] [warn=Enabled|Disabled] [list] ]