Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Advanced SR-IOV Topics: InfiniBand SR-IOV

This section describes how to identify InfiniBand physical and virtual functions as well as to correlate the Logical Domains Manager and the Oracle Solaris view of InfiniBand physical and virtual functions.

Listing InfiniBand SR-IOV Virtual Functions

The following example shows different ways to display information about the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function. A physical function name that includes the IOVIB string indicates that it is an InfiniBand SR-IOV device.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/MB/RISER0/PCIE0                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER1/PCIE1                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER2/PCIE2                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER0/PCIE3                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER1/PCIE4                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER2/PCIE5                      PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/SASHBA1                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/NET2                              PCIE   pci_0    primary  OCC
/SYS/MB/RISER0/PCIE3/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF0        VF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF1        VF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF3        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1

The ldm list-io -l command provides more detailed information about the specified physical function device, /SYS/MB/RISER1/PCIE4/IOVIB.PF0. The maxvfs value shows that the maximum number of virtual functions supported by the physical device is 64. For each virtual function that is associated with the physical function, the output shows the following:

  • Function name

  • Function type

  • Bus name

  • Domain name

  • Optional status of the function

  • Device path

This ldm list-io -l output shows that VF0 and VF1 are assigned to the primary domain and that VF2 and VF3 are assigned to the iodom1 I/O domain.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,3]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,4]

Identifying InfiniBand SR-IOV Functions

This section describes how to identify the InfiniBand SR-IOV devices.

Use the ldm list-io -l command to show the Oracle Solaris device path name that is associated with each physical function and virtual function.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,3]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,4]

Use the dladm show-phys -L command to match each IP over InfiniBand (IPoIB) instance to its physical card. For example, the following command shows which IPoIB instances use the card in slot PCIE4, which is the same card shown in the previous ldm list-io -l example.

primary# dladm show-phys -L | grep PCIE4
net5              ibp0         PCIE4/PORT1
net6              ibp1         PCIE4/PORT2
net19             ibp8         PCIE4/PORT1
net9              ibp9         PCIE4/PORT2
net18             ibp4         PCIE4/PORT1
net11             ibp5         PCIE4/PORT2

Each InfiniBand host channel adapter (HCA) device has a globally unique ID (GUID). There are also GUIDs for each port (typically there are two ports to an HCA). An InfiniBand HCA GUID uniquely identifies the adapter. The port GUID uniquely identifies each HCA port and plays a role similar to a network device's MAC address. These 16-hexadecimal digit GUIDs are used by InfiniBand management tools and diagnostic tools.

Use the dladm show-ib command to obtain GUID information about the InfiniBand SR-IOV devices. Physical functions and virtual functions for the same device have related HCA GUID values. The 11th hexadecimal digit of the HCA GUID shows the relationship between a physical function and its virtual functions. Note that leading zeros are suppressed in the HCAGUID and PORTGUID columns.

For example, physical function PF0 has two virtual functions, VF0 and VF1, which are assigned to the primary domain. The 11th hexadecimal digit of each virtual function is incremented by one from the related physical function. So, if the GUID for the PF0 is 8, the GUIDs for VF0 and VF1 will be 9 and A, respectively.

The following dladm show-ib command output shows that the net5 and net6 links belong to the physical function PF0. The net19 and net9 links belong to VF0 of the same device while the net18 and net11 links belong to VF1.

primary# dladm show-ib
LINK         HCAGUID         PORTGUID        PORT STATE  PKEYS
net6         21280001A17F56  21280001A17F58  2    up     FFFF
net5         21280001A17F56  21280001A17F57  1    up     FFFF
net19        21290001A17F56  14050000000001  1    up     FFFF
net9         21290001A17F56  14050000000008  2    up     FFFF
net18        212A0001A17F56  14050000000002  1    up     FFFF
net11        212A0001A17F56  14050000000009  2    up     FFFF

The device in the following dladm show-phys output shows the relationship between the links and the underlying InfiniBand port devices (ibpX).

primary# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
...
net6              Infiniband           up         32000  unknown   ibp1
net5              Infiniband           up         32000  unknown   ibp0
net19             Infiniband           up         32000  unknown   ibp8
net9              Infiniband           up         32000  unknown   ibp9
net18             Infiniband           up         32000  unknown   ibp4
net11             Infiniband           up         32000  unknown   ibp5

Use the ls -l command to show the actual InfiniBand port (IB port) device paths. An IB port device is a child of a device path that is shown in the ldm list-io -l output. A physical function has a one-part unit address such as pciex15b3,673c@0 while virtual functions have a two-part unit address, pciex15b3,1002@0,2. The second part of the unit address is one higher than the virtual function number. (In this case, the second component is 2, so this device is virtual function 1.) The following output shows that /dev/ibp0 is a physical function and /dev/ibp5 is a virtual function.

primary# ls -l /dev/ibp0
lrwxrwxrwx   1 root     root          83 Apr 18 12:02 /dev/ibp0 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0/hermon@0/ibport@1,0,ipib:ibp0
primary# ls -l /dev/ibp5
lrwxrwxrwx   1 root     root          85 Apr 22 23:29 /dev/ibp5 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,2/hermon@3/ibport@2,0,ipib:ibp5

You can use the OpenFabrics ibv_devices command to view the OpenFabrics device name and the node (HCA) GUID. When virtual functions are present, the Type column indicates whether the function is physical or virtual.

primary# ibv_devices
device                 node GUID            type
------              ----------------        ----
mlx4_4              0002c90300a38910         PF
mlx4_5              0021280001a17f56         PF
mlx4_0              0002cb0300a38910         VF
mlx4_1              0002ca0300a38910         VF
mlx4_2              00212a0001a17f56         VF                        
mlx4_3              0021290001a17f56         VF