x64: PCI Subsystem ID Changes in ConnectX Firmware 2.6.0 From Mellanox (6810093)

Upgrading the ConnectX firmware to version 2.6.000 or higher on a running system might cause problems in some HCAs and x64 platforms. This issue affects only Mellanox-branded HCAs. Sun-branded PCIe HCAs, EMs, NEMs, and SPARC platforms are not affected.

You might be unable to boot the system or the systrem might hang during boot. ibd (IPoverIB) instance numbers might change and prevent the system from booting and plumb ibd devices.

Workaround 1: Before rebooting the system after updating the firmware using cxflash, remove the ibd<x> instances from the /etc/path_to_inst file and /dev directory. Perform the following steps:

1. Log in as the root user. The device tree information might look like this:

   # ls -R /devices | grep 15b3 /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0 /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0/ibport@1,ffff,ipib /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0/ibport@1,ffff,ipib:ibd0 /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0/ibport@2,ffff,ipib /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0/ibport@2,ffff,ipib:ibd1 /devices/pci@0,0/pci10de,5d@e/pci15b3,634a@0:devctl

   ---

   Note –

   Device specifics vary depending on the Mellanox-branded card installed.

   ---

2. Edit the path_to_inst file. Perform the following steps:

   1. Make a backup copy of the existing path_to_inst file.

      # cp /etc/path_to_inst /etc/path_to_inst.backup

   2. Open the path_to_inst file:

      # vi /etc/path_to_inst

   3. Search for lines that contain ibd and hermon, and delete those lines.

   4. Save the changes and close the file.

3. Remove the entries in the /dev directory:

   rm /dev/ibd?*

4. Reboot the system. The system boots normally, and the corresponding device tree looks like this:

   # ls -R /devices | grep 15b3 /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0 /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0/ibport@1,ffff,ipib /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0/ibport@1,ffff,ipib:ibd0 /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0/ibport@2,ffff,ipib /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0/ibport@2,ffff,ipib:ibd1 /devices/pci@0,0/pci10de,5d@e/pci15b3,5@0:devctl

Workaround 2: If you reboot the system before updating the path_to_inst file and /dev directory, the system might hang. In such a case, perform the following steps:

1. Power off the system and remove the HCA from the bus.

2. Reboot the system without the HCA installed.

3. When system is back on, follow steps 2 through 3 listed in Workaround 1.

4. Power off the system.

5. Re-intsall the HCA. Reboot the system.

6. If the system reboots, but ibd interfaces do not automatically plumb, the /etc/hostname.ib<?> files might not be appropriate for the current device configuration. Manually rename the files to reflect the correct configuration.