C H A P T E R  8

Infiniband Host Channel Adapter Test (ibhcatest)

ibhcatest is comprised of multiple iRISC CPU cores, two 4x IB ports and integrated SerDes components. In addition, the ibhcatest external associated components include FLASH ROM and DDR memory. ibhcatest provides high speed interconnect through PCI interface to external IB fabric. Supported platforms include: two 1U and two 2U x86 AMD Opteron entry-level servers, Sun Fire V2xx, V4xx and E series high end servers.



Note - ibhcatest supports the same set of options for both SPARC and x86 platforms.



ibhcatest provides a mechanism to exercise and verify the proper operation of the Tavor chip and its associated components, such as DDR memory, Flash PROM and internal IB packet transmit/receive circuitry. The goal is to isolate single fault to the identifiable component(s).

ibhcatest supports three execution test modes in SunVTS: Connection, Exclusive and Functional. In Connection mode, the test will provide a basic sanity check. This basic sanity test is done by querying for the Tavor firmware / hardware revision and running internal loopback.

The internal loopback test is run at least once depending on the amount of time each pass takes. In Functional mode all subtests are executed according to the options selected. In Exclusive mode all subtests are executed in sequence.

Tavor supports an internal loopback mechanism which is very similar to the actual operation. The main difference is that data does not go through the integrated SerDes and the 4x IB port circuitry. Also on the receiving side, data does not get verified by the CRC algorithm. Otherwise, all other components of Tavor that involve in transmitting and receiving data packets are being exercised.

Tavor based HCA is designed to use a single, 256 MB DDR memory for data storage at run time. This data storage is intended to be used and shared by three interdependent clients: Tavor driver, firmware, and hardware. During driver initialization, predetermined data structures and data are laid out in the memory.

With no exclusive atomic access from the driver side, subsequent writes to any memory location that contain real data can cause undesirable results like a system crash. Furthermore, the data allocation size is fixed, writing to the remaining free memory does not add any value in terms of finding faults.

With these constraints in mind, the memory subtest is limited to read only operations to cover the entire DDR memory. There is no checking for data corruption and no mechanism for triggering single/double bit type of errors through writing to memory. The resulting benefit of read operations is a secondary effect that occurs in the generation of high volume PCI activities from the memory accesses. Thus the test becomes a good exerciser to bring out bus related problems.


ibhcatest Subtests


TABLE 8-1 ibhcatest Subtests

Subtest

Description

Internal Loopback Test

The HCA supports internal loopback for packets transmitted between QPs that are assigned to the same HCA port. When transmitting a packet, if it is destined to a DLID that is equivalent to the Port LID with the LMC bits masked out or the packet DLID is a multicast LID, the packet goes on the loopback path. In this latter case, the packet also is transmitted to the fabric. When a packet is looped back, it must pass the SL2VL mapping. If the mapping yields 15 or a nonoperational VL, the packet is discarded. In the inbound direction, the ICRC and VCRC checks are "blindly" passed for looped back packets.

Note that internal loopback is supported only for packets that are transmitted and received on the same port. Packets that are transmitted on one port and received on another port are transmitted to the fabric. The fabric should direct them to the destination port. This subtest uses interfaces from the Tavor driver to perform loopback testing. It is a push button type of test. Information such as data pattern for data packets, port number, CQ polling, retries between iteration, and the number of iterations for each ioctl call are passed to the driver. Once finished, status regarding the number passes completed is returned. If the number of passes does not match the number of iterations, a failure has occurred. This might happen when the number of retries is exhausted, and the last failing buffer in the retry series is returned as result. SunVTS then determines exactly what failed in the buffer and reports the failure. The options for this subtest are as follows:

  • lb=Enabled|Disabled: Turn on/off the loopback test
  • tlbport=1+2: Loopback test on Port 1 and/or 2, default is 1+2
  • data=Pattern: Specific data pattern (default patterns 0xa5a5a5a5)
  • cq=Time: Number of CQ polling time value (in microseconds per iteration); default: 55000; max: 1000000.
  • loop=Number: Number of loopback iterations for each pass; default: 200; max: 1000
  • warn=Enabled|Disabled: When enabled, prints a warning message

DDR READ Test

This subtetst is comprised of two test modes, Sequential and Random . The start and end address offsets are determined dynamically by obtaining them from the firmware. In Sequential mode, length and starting offset are instructed by the test option rdoffset and rdsz. Then the test goes through and sequentially reads data from each memory address until all memory locations are covered or the end address is reached. Each read is accomplished by an ioctl call to the driver.

The test returns pass or fail based on the completion status of each ioctl call. In the random test method, this subtest reads the number of rdsz times in a randomly generated address bound by the start and end address offset. The options for this subtest are as follows:

  • ddr=Enabled|Disabled: Turn off/off the DDR Memory test
  • rdoffset=Offset: Starting offset of DDR Memory read, default is 0x0, in hexadecimal
  • rdsz=Size: Read number of byte of DDR Memory from Offset to Max address location, default: 0x2000; max: 256 MB, in hexadecimal



caution icon

Caution - In SunVTS environment ibhcatest and nettest are mutually exclusive, nettest has higher priority if the IB port interface is plumbed up when SunVTS is invoked. These two tests can not be run at the same time at the command line, if both of these tests are invoked at the command line, ibhcatest exits gracefully if the IB port interface is plumbed up. The commands to bring down the IB daemon (ibd [IPoIB]) are as follows:
# ifconfig ibdXX down
# ifconfig ibdXX unplumb
Where XX is instance number of the interface.




ibhcatest Options


To reach the dialog box below, right-click on the test name in the System Map and select Test Parameter Options. Because graphics test can test multiple types of frame buffers, the test name that is displayed will correspond to the particular framebuffer being tested. If you do not see this test in the System Map, you might need to expand the collapsed groups, or your system may not include the device appropriate to this test. Refer to the SunVTS User's Guide. FIGURE 8-1 ibhcatest Test Parameter Options Dialog Box

 


TABLE 8-2 ibhcatest Options

Option

Description

lb=Enabled|Disabled

Turn on/off the loopback test

tlbport=1+2

Loopback test on Port 1 and/or 2, default is 1+2

data=Pattern

Specific data pattern (default patterns 0xa5a5a5a5)

cq=Time

Number of CQ polling time value (in microseconds per iteration), default: 55000; max: 1000000

loop=Number

Number of loopback iterations for each pass; default: 200; max: 1000

ddr=Enabled|Disabled

Turn on/off the DDR Memory test

rdoffset=Offset

Starting offset of DDR Memory read, default is 0x0, in hexadecimal

rdsz=Size

Read number of byte of DDR Memory from Offset to Max address location, default: 0x2000; max: 256 MB, in hexadecimal

warn=Enabled|Disabled

Print a warning message when enabled

list

Print device list, no testing when set



ibhcatest Test Modes


TABLE 8-3 ibhcatest Supported Test Modes

Test Mode

Description

Connection

Provides a basic sanity check by querying for the Tavor firmware / hardware revision and running internal loopback. The internal loopback test runs at least once depending on the amount of time each pass takes.

Exclusive

Executes all subtests sequentially.

Functional

Executes all subtests according to what is selected.



ibhcatest Command Line Syntax

ibhcatest [-scruvdtlxnf] [-p n] [-i n] [-w n] [-o [dev=text] [lb=Enabled|Disabled] [tlbport=1+2] [data=Pattern] [cq=Time] [loop=Number] [ddr=Enabled|Disabled] [rdoffset=Offset] [rdsz=Len] [warn=Enabled|Disabled] [list] ]

Example:


# ibhcatest -p 0 -svf -o lb=Enabled, tlbport=1+2, data=0xA5A5A5A5, cq=55000, loop=200, 


ddr=Enabled, rdoffset=0x0, rdsz=0x2000, warn=Enabled, dev=tavor1

TABLE 8-4 ibhcatest Command Line Syntax

Option

Description

lb=Enabled|Disabled

Turn on/off the loopback test

tlbport=1+2

Loopback test on Port 1 and/or 2, default is 1+2

data=Pattern

Specific data pattern (default patterns 0xa5a5a5a5)

cq=Time

Number of CQ polling time value (in microseconds per iteration), default: 55000; max: 1000000.

loop=Number

Number of loopback iterations for each pass; default: 200; max: 1000

ddr=Enabled|Disabled

Turn on/off the DDR Memory test

rdoffset=Offset

Starting offset of DDR Memory read, default is 0x0, in hexadecimal

rdsz=Size

Read number of byte of DDR Memory from Offset to Max address location, default: 0x2000; max: 256 MB, in hexadecimal

warn=Enabled|Disabled

Print a warning message when enabled

list

Print device list, no testing when set


 

Note - 64-bit tests are located in the /bin/64 directory, or the relative path in which you installed SunVTS. If a test is not present in this directory, then it might be available as a 32-bit test only. For more information, see 32-Bit and 64-Bit Tests.