1. Verify that there is at least one Subnet Manager active on the InfiniBand fabric.

   See Display Subnet Manager Priority, Controlled Handover State, Prefix, Management Key, and Routing Algorithm.

2. If no Subnet Manager is active, start the Subnet Manager within the gateway.

   Refer to Gateway Installation, staring the Subnet Manager.

3. If the previous steps do not rectify the situation, restart the Subnet Manager.

   See Disable the Subnet Manager and Enable the Subnet Manager.

1. Determine which links are nonoperational.

   See Display Link Status.

2. For links that are “Down”, disable and re-enable the respective ports.

   See Disable a Switch Chip Port and Enable a Switch Chip Port.

3. If the previous steps do not rectify the situation, disable the respective port.

   See Disable a Switch Chip Port.

1. If you are able to access the management controller, restart the management controller.

   See Restart the Management Controller.

2. If you are unable to access the management controller, power cycle the gateway.

   Refer to Gateway Service, removing the gateway from the rack.

   Refer to Gateway Installation, installing the gateway into the rack.

3. Reperform the firmware upgrade.

   Refer to Gateway Remote Management, upgrading the gateway firmware.

1. Determine if there are errors or problems with the InfiniBand fabric.

   See:

   • Perform Comprehensive Diagnostics for the Entire Fabric

   • Find 1x, SDR, or DDR Links in the Fabric

   • Determine Which Links Are Experiencing Significant Errors

2. Locate the affected nodes by the GUID provided in the output of the ibdiagnet command.

   See Locate a Switch Chip or Connector From the GUID and Port.

3. If the problem is at a cable connection, swap the suspect cable with a known good cable or reconnect the cable to a known good remote port and repeat Step 1.

   Refer to Gateway Service, servicing data cables.

4. If the problem still remains at the cable connection, disable and re-enable the respective port and repeat Step 1.

   See Disable or Enable an External Port.

Temporary solution:

• If the problem still remains, disable the affected port.

  See Disable or Enable an External Port.

Permanent solution:

• If the problem still remains, replace the affected component or the gateway.

  Refer to Gateway Service, servicing data cables.

  Refer to remote port's documentation for replacement procedures.

  Refer to Gateway Service, removing the gateway from the rack.

  Refer to Gateway Installation, installing the gateway into the rack.

1. Disconnect and properly reconnect both ends of the respective InfiniBand cable.

   Refer to Gateway Service, servicing the data cables.

2. If the LED is still blinking, determine the significance of the errors through use of the ibdiagnet command.

   See Determine Which Links Are Experiencing Significant Errors.

3. Determine which connectors map to the affected link by deconstructing the node's GUID and port.

   See Locate a Switch Chip or Connector From the GUID and Port.

4. If some of the links are running at 1x or SDR, use that situation elsewhere in this table to rectify the problem.

5. Disable and re-enable the respective ports.

   See Disable or Enable an External Port.

6. If the errors are still significant, swap the cable with a known good one or reconnect the cable to a known good remote port, and repeat from Step 2.

7. Depending upon what does or does not rectify the problem, replace that component.

   Refer to Gateway Service, servicing the data cables.

   Refer to remote port's documentation for replacement procedures.

1. Identify the suspect links using the ibdiagnet command.

   See Find 1x, SDR, or DDR Links in the Fabric. Look for text like this:

   -W- link with SPD=2.5 found at direct path "1,19"

   From: a Switch PortGUID=0x00066a00d80001dd Port=19

   To: a Switch PortGUID=0x00066a00d80001dd Port=24

2. Determine which connectors map to the affected link by deconstructing the node's GUID and port.

   See Locate a Switch Chip or Connector From the GUID and Port.

3. Verify the cable connection at both ends.

   Refer to Gateway Service, servicing the data cables.

4. Disable and re-enable the respective ports.

   See Disable or Enable an External Port.

5. If the previous steps do not rectify the problem, disable the port.

   See Disable or Enable an External Port.

For a permanent solution:

1. Perform the steps for a temporary solution, Steps 1 to Step 4.

2. Swap the cable with a known good cable or reconnect the cable to a known good remote port, and repeat from Step 1.

3. Depending upon what does or does not rectify the problem, replace that component or the gateway.

   Refer to Gateway Service, servicing the data cables.

   Refer to the remote port's documentation for replacement procedures.

   Refer to Gateway Service, removing the gateway from the rack.

   Refer to Gateway Installation, installing the gateway into the rack.

1. Clear the error counters.

   See Clear Data and Error Counters.

2. Start a fabric stress test.

3. Identify the suspect links using the ibdiagnet command.

   See Determine Which Links Are Experiencing Significant Errors. Look for text like this:

   -W- lid=0x0006 guid=0x0021283a8816c0a0 dev=48438 Port=34

   Performance Monitor counter : Value

   link_recovery_error_counter : 0x1

   symbol_error_counter : 0x25 (Increase by 3 during ibdiagnet)

4. For links that are experiencing recovery errors or substantial symbol errors, refer to other parts of this table to help identify the cause and rectify the problem.

1. You can find the location of a node in the gateway by deconstructing the node's GUID and port.

   See Locate a Switch Chip or Connector From the GUID and Port.

2. Use the dcsport command to provide port-to-connector and connector-to-port mapping.

   See Display the Switch Chip Port to QSFP Connector Mapping.