Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Creating and Destroying InfiniBand Virtual Functions

How to Create an InfiniBand Virtual Function

This procedure describes how to create an InfiniBand SR-IOV virtual function.

  1. Initiate a delayed reconfiguration on the root domain.
    primary# ldm start-reconf root-domain-name
  2. Enable I/O virtualization by setting iov=on.

    Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.

    primary# ldm set-io iov=on bus
  3. Create one or more virtual functions that are associated with the physical functions from that root domain.
    primary# ldm create-vf pf-name

    You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 10, Creating Multiple SR-IOV Ethernet Virtual Functions and the ldm(8) man page.

  4. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y
Example 18  Creating an InfiniBand Virtual Function

    The following example shows information about the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function:

  • This physical function is in PCIE slot 4.

  • The IOVIB string indicates that the physical function is an InfiniBand SR-IOV device.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
/SYS/MB/RISER0/PCIE0                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER1/PCIE1                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER2/PCIE2                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER0/PCIE3                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER1/PCIE4                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER2/PCIE5                      PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/SASHBA1                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/NET2                              PCIE   pci_0    primary  OCC
/SYS/MB/RISER0/PCIE3/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_0    primary

The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that are supported by the device.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64

The following example shows how to create a static virtual function. First, initiate a delayed reconfiguration on the primary domain and enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has been assigned already to the primary root domain, use the ldm set-io command to enable I/O virtualization.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------

Now, use the ldm create-vf command to create a virtual function from the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function.

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0

Note that you can create more than one virtual function during the same delayed reconfiguration. The following command creates a second virtual function:

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Shutdown started.

Changing to init state 6 - please wait
...

How to Destroy an InfiniBand Virtual Function

This procedure describes how to destroy an InfiniBand SR-IOV virtual function.

A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can be destroyed only in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver.

An attempt to use a dynamic SR-IOV remove operation on an InfiniBand device results in confusing and inappropriate error messages.

Dynamic SR-IOV remove operations are not supported for InfiniBand devices.

Remove InfiniBand virtual functions by performing one of the following procedures:

  1. Initiate a delayed reconfiguration on the root domain.
    primary# ldm start-reconf root-domain-name
  2. Destroy one or more virtual functions that are associated with the physical functions from that root domain.
    primary# ldm destroy-vf vf-name

    You can run this command for each virtual function that you want to destroy. You can also use the –n option to destroy more than one virtual function from the same physical function in a single command. See Example 12, Destroying Multiple Ethernet SR-IOV Virtual Functions and the ldm(8) man page.

  3. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y
Example 19  Destroying an InfiniBand Virtual Function

The following example shows how to destroy a static InfiniBand virtual function, /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1.

The ldm list-io command shows information about the buses, physical functions, and virtual functions.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN  STATUS
----                                      ----   ---      ------  ------
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0

You can obtain more details about the physical function and related virtual functions by using the ldm list-io -l command.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]

A virtual function can be destroyed only if it is unassigned to a domain. The DOMAIN column of the ldm list-io -l output shows the name of any domain to which a virtual function is assigned. Also, virtual functions must be destroyed in the reverse order of their creation. Therefore, in this example, you must destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 virtual function before you can destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 virtual function.

After you identify the proper virtual function, you can destroy it. First, initiate a delayed reconfiguration.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm destroy-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------

You can issue more than one ldm destroy-vf command while in delayed reconfiguration. Thus, you could also destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0.

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Shutdown started.

Changing to init state 6 - please wait
...