Identifying InfiniBand SR-IOV Functions
This section describes how to identify the InfiniBand SR-IOV devices.
Use the ldm list-io -l
command to show the Oracle Solaris device path name that is associated with each physical function and virtual function.
primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0 PF pci_0 primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 VF pci_0 primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 VF pci_0 primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 VF pci_0 iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,3]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3 VF pci_0 iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,4]
Use the dladm show-phys -L
command to match each IP over InfiniBand (IPoIB) instance to its physical card. For example, the following command shows which IPoIB instances use the card in slot PCIE4
, which is the same card shown in the previous ldm list-io -l
example.
primary# dladm show-phys -L | grep PCIE4
net5 ibp0 PCIE4/PORT1
net6 ibp1 PCIE4/PORT2
net19 ibp8 PCIE4/PORT1
net9 ibp9 PCIE4/PORT2
net18 ibp4 PCIE4/PORT1
net11 ibp5 PCIE4/PORT2
Each InfiniBand host channel adapter (HCA) device has a globally unique ID (GUID). There are also GUIDs for each port (typically there are two ports to an HCA). An InfiniBand HCA GUID uniquely identifies the adapter. The port GUID uniquely identifies each HCA port and plays a role similar to a network device's MAC address. These 16-hexadecimal digit GUIDs are used by InfiniBand management tools and diagnostic tools.
Use the dladm show-ib
command to obtain GUID information about the InfiniBand SR-IOV devices. Physical functions and virtual functions for the same device have related HCA GUID values. The 11th hexadecimal digit of the HCA GUID shows the relationship between a physical function and its virtual functions. Note that leading zeros are suppressed in the HCAGUID and PORTGUID columns.
For example, physical function PF0
has two virtual functions, VF0
and VF1
, which are assigned to the primary
domain. The 11th hexadecimal digit of each virtual function is incremented by one from the related physical function. So, if the GUID for the PF0
is 8
, the GUIDs for VF0
and VF1
will be 9
and A
, respectively.
The following dladm show-ib
command output shows that the net5
and net6
links belong to the physical function PF0
. The net19
and net9
links belong to VF0
of the same device while the net18
and net11
links belong to VF1
.
primary# dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS net6 21280001A17F56 21280001A17F58 2 up FFFF net5 21280001A17F56 21280001A17F57 1 up FFFF net19 21290001A17F56 14050000000001 1 up FFFF net9 21290001A17F56 14050000000008 2 up FFFF net18 212A0001A17F56 14050000000002 1 up FFFF net11 212A0001A17F56 14050000000009 2 up FFFF
The device in the following dladm show-phys
output shows the relationship between the links and the underlying InfiniBand port devices (ibp
X).
primary# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
...
net6 Infiniband up 32000 unknown ibp1
net5 Infiniband up 32000 unknown ibp0
net19 Infiniband up 32000 unknown ibp8
net9 Infiniband up 32000 unknown ibp9
net18 Infiniband up 32000 unknown ibp4
net11 Infiniband up 32000 unknown ibp5
Use the ls -l
command to show the actual InfiniBand port (IB port) device paths. An IB port device is a child of a device path that is shown in the ldm list-io -l
output. A physical function has a one-part unit address such as pciex15b3,673c@0
while virtual functions have a two-part unit address, pciex15b3,1002@0,2
. The second part of the unit address is one higher than the virtual function number. (In this case, the second component is 2
, so this device is virtual function 1
.) The following output shows that /dev/ibp0
is a physical function and /dev/ibp5
is a virtual function.
primary# ls -l /dev/ibp0 lrwxrwxrwx 1 root root 83 Apr 18 12:02 /dev/ibp0 -> ../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0/hermon@0/ibport@1,0,ipib:ibp0 primary# ls -l /dev/ibp5 lrwxrwxrwx 1 root root 85 Apr 22 23:29 /dev/ibp5 -> ../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,2/hermon@3/ibport@2,0,ipib:ibp5
You can use the OpenFabrics ibv_devices
command to view the OpenFabrics device name and the node (HCA) GUID. When virtual functions are present, the Type column indicates whether the function is physical or virtual.
primary# ibv_devices
device node GUID type
------ ---------------- ----
mlx4_4 0002c90300a38910 PF
mlx4_5 0021280001a17f56 PF
mlx4_0 0002cb0300a38910 VF
mlx4_1 0002ca0300a38910 VF
mlx4_2 00212a0001a17f56 VF
mlx4_3 0021290001a17f56 VF