Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Using an SR-IOV Virtual Function to Create an I/O Domain

The following procedure explains how to create an I/O domain that includes PCIe SR-IOV virtual functions.

How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It

Plan ahead to minimize the number of reboots of the root domain, which minimizes downtime.

Before You Begin

Before you begin, ensure that you have enabled I/O virtualization for the PCIe bus that is the parent of the physical function from which you create virtual functions. See How to Enable I/O Virtualization for a PCIe Bus.

  1. Identify an SR-IOV physical function to share with an I/O domain that uses the SR-IOV feature.
    primary# ldm list-io
  2. Create one or more virtual functions for the physical function.
    primary# ldm create-vf pf-name

    You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 10, Creating Multiple SR-IOV Ethernet Virtual Functions and the ldm(8) man page.


    Note - This command fails if other virtual functions have already been created from the associated physical function and if any of those virtual functions are bound to another domain.
  3. View the list of available virtual functions on the root domain.
    primary# ldm list-io
  4. Assign the virtual function that you created in Step 2 to its target I/O domain.
    primary# ldm add-io vf-name domain-name

    Note - If the OS in the target I/O domain does not support dynamic SR-IOV, you must use the static method. See Static SR-IOV.
  5. Verify that the virtual function is available on the I/O domain.

    The following Oracle Solaris 11 command shows the availability of the virtual function:

    guest# dladm show-phys
Example 16  Dynamically Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It

The following dynamic example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.

    This example assumes that the following circumstances are true:

  • The OS on the primary domain supports dynamic SR-IOV operations

  • The pci_0 bus is assigned to the primary domain and has been initialized for I/O virtualization operations

  • The /SYS/MB/NET0/IOVNET.PF0 physical function belongs to the pci_0 bus

  • The /SYS/MB/NET0/IOVNET.PF0 physical function does not have any existing virtual functions assigned to domains

  • The ldg1 domain is active and booted and its OS supports dynamic SR-IOV operations

Create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain.

primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

The following command shows that the virtual function has been added to the ldg1 domain.

primary# ldm list-io
NAME                          TYPE  BUS      DOMAIN    STATUS
----                          ----  ---      ------    ------
/SYS/PM0/CMP0/NIU_CORE        NIU   niu_0    primary
/SYS/PM0/CMP1/NIU_CORE        NIU   niu_1    primary
/SYS/PM0/CMP0/PEX             BUS   pci_0    primary   IOV
/SYS/PM0/CMP1/PEX             BUS   pci_1
/SYS/MB/PCIE0                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE2                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE4                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE6                 PCIE  pci_0    primary   EMP
/SYS/MB/PCIE8                 PCIE  pci_0    primary   EMP
/SYS/MB/SASHBA                PCIE  pci_0    primary   OCC
/SYS/MB/NET0                  PCIE  pci_0    primary   OCC
/SYS/MB/PCIE1                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE3                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE5                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE7                 PCIE  pci_1    primary   EMP
/SYS/MB/PCIE9                 PCIE  pci_1    primary   EMP
/SYS/MB/NET2                  PCIE  pci_1    primary   OCC
/SYS/MB/NET0/IOVNET.PF0       PF    pci_0    primary
/SYS/MB/NET0/IOVNET.PF1       PF    pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0      PF    pci_1    primary
/SYS/MB/PCIE5/IOVNET.PF1      PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF0       PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF1       PF    pci_1    primary
/SYS/MB/NET0/IOVNET.PF0.VF0   VF    pci_0    ldg1
Example 17  Statically Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It

The following static example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.

    This example assumes that the following circumstances are true:

  • The OS on the primary domain does not support dynamic SR-IOV operations

  • The pci_0 bus is assigned to the primary domain and has not been initialized for I/O virtualization operations

  • The /SYS/MB/NET0/IOVNET.PF0 physical function belongs to the pci_0 bus

  • The /SYS/MB/NET0/IOVNET.PF0 physical function does not have any existing virtual functions assigned to domains

  • The ldg1 domain is active and booted and its OS does not support dynamic SR-IOV operations

  • The ldg1 domain has the auto-boot? property set to true so that the domain boots automatically when the domain is started

First, initiate a delayed reconfiguration on the primary domain, enable I/O virtualization, and create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0

------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Next, shut down the primary domain.

primary# shutdown -i6 -g0 -y

Stop the ldg1 domain, add the virtual function, and start the domain.

primary# ldm stop ldg1
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
primary# ldm start-domain ldg1

The following command shows that the virtual function has been added to the ldg1 domain.

primary# ldm list-io
NAME                          TYPE  BUS      DOMAIN    STATUS
----                          ----  ---      ------    ------
/SYS/PM0/CMP0/NIU_CORE        NIU   niu_0    primary
/SYS/PM0/CMP1/NIU_CORE        NIU   niu_1    primary
/SYS/PM0/CMP0/PEX             BUS   pci_0    primary   IOV
/SYS/PM0/CMP1/PEX             BUS   pci_1
/SYS/MB/PCIE0                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE2                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE4                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE6                 PCIE  pci_0    primary   EMP
/SYS/MB/PCIE8                 PCIE  pci_0    primary   EMP
/SYS/MB/SASHBA                PCIE  pci_0    primary   OCC
/SYS/MB/NET0                  PCIE  pci_0    primary   OCC
/SYS/MB/PCIE1                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE3                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE5                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE7                 PCIE  pci_1    primary   EMP
/SYS/MB/PCIE9                 PCIE  pci_1    primary   EMP
/SYS/MB/NET2                  PCIE  pci_1    primary   OCC
/SYS/MB/NET0/IOVNET.PF0       PF    pci_0    primary
/SYS/MB/NET0/IOVNET.PF1       PF    pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0      PF    pci_1    primary
/SYS/MB/PCIE5/IOVNET.PF1      PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF0       PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF1       PF    pci_1    primary
/SYS/MB/NET0/IOVNET.PF0.VF0   VF    pci_0    ldg1