How to Create an InfiniBand Virtual Function

This procedure describes how to create an InfiniBand SR-IOV virtual function.

1. Initiate a delayed reconfiguration on the root domain.

   primary# ldm start-reconf root-domain-name

2. Enable I/O virtualization by setting iov=on.

   Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.

   primary# ldm set-io iov=on bus

3. Create one or more virtual functions that are associated with the physical functions from that root domain.

   primary# ldm create-vf pf-name

   You can run this command for each virtual function that you want to create. You can also use the -n option to create more than one virtual function from the same physical function in a single command. See Creating Multiple SR-IOV Ethernet Virtual Functions and the ldm(8) man page.

4. Reboot the root domain.

   Run one of the following commands:

   • Reboot the non-primary root domain.

     primary# ldm stop-domain -r root-domain

   • Reboot the primary root domain.

     primary# shutdown -i6 -g0 -y

Example 8-14 Creating an InfiniBand Virtual Function

The following example shows information about the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function:

• This physical function is in PCIE slot 4.

• The IOVIB string indicates that the physical function is an InfiniBand SR-IOV device.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
/SYS/MB/RISER0/PCIE0                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER1/PCIE1                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER2/PCIE2                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER0/PCIE3                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER1/PCIE4                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER2/PCIE5                      PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/SASHBA1                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/NET2                              PCIE   pci_0    primary  OCC
/SYS/MB/RISER0/PCIE3/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_0    primary

The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that are supported by the device.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64

The following example shows how to create a static virtual function. First, initiate a delayed reconfiguration on the primary domain and enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has been assigned already to the primary root domain, use the ldm set-io command to enable I/O virtualization.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------

Now, use the ldm create-vf command to create a virtual function from the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function.

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0

Note that you can create more than one virtual function during the same delayed reconfiguration. The following command creates a second virtual function:

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Shutdown started.

Changing to init state 6 - please wait
...