15.2 Control the InfiniBand Fabric

This section contains the following topics:

15.2.1 Clear Error Counters

If you are troubleshooting a port, the perfquery command provides counters of errors occurring at that port. To determine if the problem has been resolved, you can reset all of the error counters to 0 with the ibclearerrors command.

On the command-line interface (CLI), run the following command:

# ibclearerrors

The output is displayed, as in the following example:

## Summary: 5 nodes cleared 0 errors
#

15.2.2 Clear Data Counters

When you are optimizing the InfiniBand fabric for performance, you might want to know how the throughput increases or decreases according to changes you are making to the fabric and Subnet Manager. The ibclearcounters command enables you to reset the data counters for all ports to 0.

On the command-line interface (CLI), run the following command:

# ibclearcounters

The output is displayed, as in the following example:

## Summary: 5 nodes cleared 0 errors
#

15.2.3 Reset a Port

You might need to reset a port to determine its functionality.

On the command-line interface (CLI), run the following command:

# ibportstate lid port reset

where lid is the LID of the node in the fabric, and port is the port of the node.

For example:

# ibportstate 15 23 reset
Initial PortInfo:
# Port info: Lid 15 port 23
LinkState:.......................Down
PhysLinkState:...................Disabled
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
After PortInfo set:
# Port info: Lid 15 port 23
LinkState:.......................Down
PhysLinkState:...................Disabled
After PortInfo set:
# Port info: Lid 15 port 23

LinkState:.......................Down
PhysLinkState:...................PortConfigurationTraining
#

15.2.4 Set Port Speed

You can manually set the speed of a single port to help determine symbol error generation. The ibportstate command can set the speed to 2.5, 5.0, or 10.0 GB/sec.

On the command-line interface (CLI), run the following command:

# ibportstate lid port speed <value>

where lid is the LID of the node in the fabric, port is the port of the node, and <value> is the speed you want to set.

Note:

Adding speed values enables either speed. For example, speed 7 is 2.5, 5.0, and 10.0 GB/sec.

For example:

# ibportstate 15 23 speed 1
Initial PortInfo:
# Port info: Lid 15 port 23
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
After PortInfo set:
# Port info: Lid 15 port 23
LinkSpeedEnabled:................2.5 Gbps
# ibportstate 15 23 speed 7
Initial PortInfo:
# Port info: Lid 15 port 23
LinkSpeedEnabled:................2.5 Gbps
After PortInfo set:
# Port info: Lid 15 port 23
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
#

15.2.5 Disable a Port

If a port is found to be problematic due to a bad cable connection or a physical damage to the connectors, you can disable the port.

On the command-line interface (CLI), run the following command:

# disableswitchport [--reason=reason] connector|ibdev port

where reason is the reason for disabling the port, Blacklist or Partition. connector is the number of the QSFP connector (0A–15B). ibdev is the InfiniBand device name (Switch, Bridge-0-0, Bridge-0-1, Bridge-1-0, Bridge-1-1). port is the number of the port (1–36).

This hardware command disables a QSFP connector and port on the switch chip or a port on the BridgeX chips. The command addresses either the connector or the port on the switch chip or the BridgeX port.

The --reason option enables you to use a passphrase to lock the state of the port:

  • Blacklist – A connector and port pair are identified as being inaccessible because of unreliable operation.

  • Partition – A connector and port pair are identified as being isolated from the InfiniBand fabric.

Both the Blacklist and Partition passphrases survive reboot. You unlock these passphrases using the enableswitchport command with the --reason option.

Note:

State changes made with the ibportstate command are not recognized by the disableswitchport, enableswitchport, or listlinkup commands.

The following example shows how to disable and blacklist connector 14A with the disableswitchport command.:

# disableswitchport --reason=Blacklist 14A
Disable Switch port 7 reason: Blacklist
Initial PortInfo:
# Port info: DR path slid 65535; dlid 65535; 0 port 7
LinkState:.......................Down
PhysLinkState:...................Polling
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
After PortInfo set:
# Port info: DR path slid 65535; dlid 65535; 0 port 7
LinkState:.......................Down
PhysLinkState:...................Disabled
#

Note:

After fixing the cable connection or any connector problems, you should enable the port.

15.2.6 Enable a Port

After fixing any connection- or connector-related problem related to a port, you should enable the port with the enableswitchport command.

On the command-line interface (CLI), run the following command:

enableswitchport [--reason=reason] connector|ibdev port

where reason is the reason for disabling the port, connector is the number of the QSFP connector (0A–15B), ibdev is the InfiniBand device name (Switch, Bridge-0-0, Bridge-0-1, Bridge-1-0, Bridge-1-1), and port is the number of the port (1–36).

For example:

# enableswitchport --reason=Blacklist 14A
Enable Switch port 7
Initial PortInfo:
# Port info: DR path slid 65535; dlid 65535; 0 port 7
LinkState:.......................Down
PhysLinkState:...................Disabled
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
LinkSpeedActive:.................2.5 Gbps
After PortInfo set:
# Port info: DR path slid 65535; dlid 65535; 0 port 7
LinkState:.......................Down
PhysLinkState:...................Polling
#