Using PCIe SR-IOV Virtual Functions

Support for the Peripheral Component Interconnect Express (PCIe) single root I/O virtualization (SR-IOV) feature has been added starting with the Oracle VM Server for SPARC 2.2 release.

SR-IOV Overview

The SR-IOV implementation is based on version 1.1 of the standard as defined by the PCI-SIG. The SR-IOV standard enables the efficient sharing of PCIe devices among virtual machines and is implemented in the hardware to achieve I/O performance that is comparable to native performance. The SR-IOV specification defines a new standard wherein new devices that are created enable the virtual machine to be directly connected to the I/O device.

A single I/O resource, which is known as a physical function, can be shared by many virtual machines. The shared devices provide dedicated resources and also use shared common resources. In this way, each virtual machine has access to unique resources. Therefore, a PCIe device, such as an Ethernet port, that is SR-IOV-enabled with appropriate hardware and OS support can appear as multiple, separate physical devices, each with its own PCIe configuration space.

For more information about SR-IOV, see the PCI-SIG web site.

The following figure shows the relationship between virtual functions and a physical function in an I/O domain.

Figure 6-3 Using Virtual Functions and a Physical Function in an I/O Domain

image:Diagram shows how to use virtual functions and physical functions in an I/O domain.

SR-IOV has the following function types:

Physical function – A PCI function that supports the SR-IOV capabilities as defined by the SR-IOV specification. A physical function contains the SR-IOV capability structure and manages the SR-IOV functionality. Physical functions are fully featured PCIe functions that can be discovered, managed, and manipulated like any other PCIe device. Physical functions can be used to configure and control a PCIe device.
Virtual function – A PCI function that is associated with a physical function. A virtual function is a lightweight PCIe function that shares one or more physical resources with the physical function and with virtual functions that are associated with that physical function. Unlike a physical function, a virtual function can only configure its own behavior.

Each SR-IOV device can have a physical function and each physical function can have up to 64,000 virtual functions associated with it. This number is dependent on the particular SR-IOV device. The virtual functions are created by the physical function.

After SR-IOV is enabled in the physical function, the PCI configuration space of each virtual function can be accessed by the bus, device, and function number of the physical function. Each virtual function has a PCI memory space, which is used to map its register set. The virtual function device drivers operate on the register set to enable its functionality and the virtual function appears as an actual PCI device. After creation, you can directly assign a virtual function to an I/O domain. This capability enables the virtual function to share the physical device and to perform I/O without CPU and hypervisor software overhead.

The following benefits are associated with devices that have SR-IOV capabilities:

Higher performance and reduced latency – Direct access to hardware from a virtual machines environment
Cost reduction – Capital and operational expenditure savings, which include:
- Power savings
- Reduced adapter count
- Less cabling
- Fewer switch ports

SR-IOV Hardware and Software Requirements

Support for the PCIe SR-IOV feature has been added starting with the Oracle VM Server for SPARC 2.2 release. For information about the required hardware, software, and firmware versions, see PCIe SR-IOV Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.

To create and destroy virtual functions in the SR-IOV physical function devices, you must first enable I/O virtualization on that PCIe bus.

You can use the ldm set-io or ldm add-io command to set the iov property to on. You can also use the ldm add-domain or ldm set-domain command to set the rc-add-policy property to iov. See the ldm(1M) man page.

Rebooting the root domain affects SR-IOV, so carefully plan your direct I/O configuration changes to maximize the SR-IOV related changes to the root domain and to minimize root domain reboots.

Current SR-IOV Feature Limitations

Note - To minimize reboots, perform multiple operations within the same delayed reconfiguration.

The SR-IOV feature has the following limitations in this release:

Migration is disabled for any domain that has one or more virtual functions assigned to it.
You can only destroy the last virtual function that was created for a physical function. So, if you create three virtual functions, the first virtual function that you can destroy must be the third one.
Only Ethernet SR-IOV cards are supported.
The SR-IOV feature is enabled only for the SR-IOV cards that are installed on the primary domain. If an SR-IOV card is assigned to a domain, by using either PCIe bus assignment or the Direct I/O (DIO) feature, the SR-IOV feature is not enabled for that card.
You can enable VLAN configurations of virtual functions by setting either the pvid or the vid property. You cannot set both of these virtual function properties simultaneously.

Planning for the Use of PCIe SR-IOV Virtual Functions

Before you create and destroy virtual functions, plan ahead to determine the virtual functions that you want to use in your configuration. When you create and destroy virtual functions, the primary domain must be rebooted. Such a reboot negatively affects all of the I/O domains that have either PCIe endpoints or the SR-IOV virtual functions configured. So, it is important to minimize the number of primary domain reboots. Determine the number of virtual functions that you require from the various SR-IOV devices to satisfy your current and future configuration needs.

For information about I/O domains, see General Guidelines for Creating an I/O Domain.

Use the following general steps to plan and perform SR-IOV virtual function configuration and assignment:

Determine the PCIe SR-IOV physical functions that are available on your system and which ones are best suited to your needs.

Use the following commands to identify the required information:
ldm list-io

Identifies the available SR-IOV physical function devices.

prtdiag -v

Identifies which PCIe SR-IOV cards and on-board devices are available.

ldm list-io -l pf-name

Identifies additional information about a specified physical function, such as the maximum number of virtual functions that are supported by the device.

ldm list-io -d pf-name

Identifies the device-specific properties that are supported by the device. See Advanced SR-IOV Topics.
Initiate a delayed reconfiguration on the root domain.
```
primary# ldm start-reconf primary
```
Enable I/O virtualization operations for a PCIe bus.
```
primary# ldm set-io iov=on bus
```
Create the required number of virtual functions on the specified SR-IOV physical function.

Use the following command to create the virtual function:
```
primary# ldm create-vf pf-name
```
Use the ldm create-vf command to set device-specific and network-specific properties of a virtual function. The unicast-slots property is device-specific. The mac-addr, alt-mac-addrs, mtu, pvid, and vid properties are network-specific.

Note that the mac-addr, alt-mac-addrs, and mtu network-specific properties can be changed as follows:
- When the virtual function is assigned to the primary domain: You must initiate a delayed reconfiguration.
- When the virtual function is assigned to an active I/O domain: A property change request is rejected because the change must be made when the owning domain is in the inactive or bound state.
- When the virtual function is assigned to a non-primary domain and a delayed reconfiguration is already in effect: A property change request fails with an error message.
The pvid and vid network-specific properties can be changed without restriction.

During a delayed reconfiguration, you can create more virtual functions and perform just one primary domain reboot to make the changes take effect. You do not need to reboot the primary domain after you create each virtual function.

A particular SR-IOV physical function might support many virtual functions. Create only the virtual functions that you require. For the recommended maximum number of configurations, see Advanced SR-IOV Topics.
Use the ldm add-config command to save the configuration to the SP.
Reboot the root domain to create the virtual function.
An active domain must be stopped before you use the ldm add-io command to assign a virtual function to it. Minimize I/O domain downtime by collectively making all I/O domain changes together. This method enables you to reduce the number of primary domain reboots that is required to set up such configurations.
Boot the I/O domains and configure the virtual functions as if they were any other network devices.

For information about virtual function limitations, see Advanced SR-IOV Topics.

Creating, Modifying, and Destroying Virtual Functions

This section describes how to create, modify, and destroy virtual functions.

How to Create a Virtual Function

Identify the physical function device.
```
primary# ldm list-io
```
Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.
Initiate a delayed reconfiguration on the control domain.
```
primary# ldm start-reconf primary
```
Enable I/O virtualization by setting iov=on only if it is not already enabled for the bus that has the physical function.
```
primary# ldm set-io iov=on bus
```
Create a virtual function from a physical function.
```
primary# ldm create-vf [mac-addr=num] [alt-mac-addrs=[auto|num1,[auto|num2,...]]]
  [pvid=pvid] [vid=vid1,vid2,...] [mtu=size] [name=value...] pf-name
```
Note - The MAC address is automatically allocated for network devices.

You can use either the path name or the pseudonym name to specify virtual functions. However, it is best to use the pseudonym name.

Example 6-1 Creating a Virtual Function

The following example shows information about the the physical function, /SYS/MB/NET0/IOVNET.PF0:

This physical function is from an on-board NET0 network device.
The IOVNET string indicates that the physical function is a network SR-IOV device.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary  
pci_1                                     BUS    pci_1    primary  
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that is supported by the device.

primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0]
    maxvfs = 7

The following examples show how to create a virtual function:

This example creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0

primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

This example creates a virtual function while setting the mac-addr property to 00:14:2f:f9:14:c0 and the vid property to VLAN IDs 2 and 3.
```
primary# ldm create-vf mac-addr=00:14:2f:f9:14:c0 vid=2,3 /SYS/MB/NET0/IOVNET.PF0
```
This example creates a virtual function that has two alternate MAC addresses. One MAC address is automatically allocated, and the other is explicitly specified as 00:14:2f:f9:14:c2.
```
primary# ldm create-vf alt-mac-addrs=auto,00:14:2f:f9:14:c2 /SYS/MB/NET0/IOVNET.PF0
```

How to Modify a Virtual Function

The ldm set-io bus command modifies the current configuration of a virtual function by changing the property values or by setting new properties. This command can modify both the network-specific properties and the device-specific properties. For information about device-specific properties, see Advanced SR-IOV Topics.

You can use the ldm set-io command to modify the following properties:

mac-addr, alt-mac-addrs, and mtu

To change these virtual function properties, first stop the domain that owns the virtual function and initiate a delayed reconfiguration. You must reboot the domain to make the changes take effect.
pvid and vid

You can dynamically change these properties while the virtual functions are assigned to a domain. Note that doing so might result in a change to the network traffic of an active virtual function. Namely, setting the pvid property enables a transparent VLAN. Setting the vid property to specify VLAN IDs permits VLAN traffic to those specified VLANs.
Device-specific properties

Use the ldm list-io -d pf-name command to view the list of valid device-specific properties. You can modify these properties for both the physical function and the virtual function. You must initiate a delayed reconfiguration and reboot the primary domain to make the changes take effect. For more information about device-specific properties, see Advanced SR-IOV Topics.

Modify a virtual function.

primary# ldm set-io name=value [name=value...] vf-name

Example 6-2 Modifying a Virtual Function

The following examples show how to use the ldm set-io command to set properties on a virtual function:

The following example modifies the specified virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, to be part of VLAN IDs 2, 3, and 4:
```
primary# ldm set-io vid=2,3,4 /SYS/MB/NET0/IOVNET.PF0.VF0
```
Note that this command dynamically changes the VLAN association for a virtual function. To use these VLANs, the VLAN interfaces in the I/O domains must be configured by using the appropriate Oracle Solaris OS networking commands.
The following example sets the pvid property value to 2 for the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function, which transparently makes the virtual function part of VLAN 2. Namely, the virtual function will not view any tagged VLAN traffic.
```
primary# ldm set-io pvid=2 /SYS/MB/NET0/IOVNET.PF0.VF0
```
The following example assigns three automatically allocated alternate MAC addresses to a virtual function. The alternate addresses enable the creation of Oracle Solaris 11 virtual network interface cards (VNICs) on top of a virtual function. Note that to use VNICs, you must run the Oracle Solaris 11 OS in the domain.

Note - Before you run this command, stop the domain that owns the virtual function.
```
primary# ldm set-io alt-mac-addrs=auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0
```

The following example sets the device-specific unicast-slots property to 12 for the specified virtual function. To find the device-specific properties that are valid for a physical function, use the ldm list-io -d pf-name command.

primary# ldm set-io unicast-slots=12 /SYS/MB/NET0/IOVNET.PF0.VF0

All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

How to Destroy a Virtual Function

A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can only be destroyed in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver. You must initiate a delayed reconfiguration and reboot the domain to make the change take effect.

Initiate a delayed reconfiguration on the root domain.
```
primary# ldm start-reconf primary
```
Destroy a virtual function.
```
primary# ldm destroy-vf vf-name
```

Example 6-3 Destroying a Virtual Function

The following example shows how to destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function:

primary# ldm destroy-vf  /SYS/MB/NET0/IOVNET.PF0.VF0

All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

Adding and Removing Virtual Functions on I/O Domains

How to Add a Virtual Function to an I/O Domain

The following command adds a virtual function to a logical domain:

ldm add-io vf-name ldom

vf-name is the pseudonym name or the path name of the virtual function. It is best to use the pseudonym name. ldom specifies the name of the domain to which you add the virtual function. The specified guest must be in the inactive or bound state. If you specify the primary domain, you must initiate a delayed reconfiguration.

Add a virtual function.
```
primary# ldm add-io vf-name ldom
```
The device path name for the virtual function in the domain is the path shown in the list-io -l output.

Example 6-4 Adding a Virtual Function

The following example shows how to add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain. To succeed, the specified domain must be in the inactive or bound state. If the domain is the primary domain, you must initiate a delayed reconfiguration.

primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

If the command succeeds, the virtual function is added to the ldg1 domain. If ldg1 is already bound (or is subsequently bound), the domain can be started and the guest OS can use the added virtual function for I/O operations.

How to Remove a Virtual Function From an I/O Domain

The following command removes an SR-IOV virtual function from a logical domain:

ldm remove-io vf-name ldom

vf-name is the pseudonym name or the path name of the virtual function. It is best to use the device pseudonym. ldom specifies the name of the domain from which you remove the virtual function. The specified guest must be in the inactive or bound state. If you specify the primary domain, you must initiate a delayed reconfiguration.

Note - Before removing the virtual function from the domain, ensure that it is not critical for booting that domain.

Remove a virtual function.
```
primary# ldm rm-io vf-name ldom
```

Example 6-5 Removing a Virtual Function

The following example shows how to remove the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function from the ldg1 domain:

primary# ldm rm-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

If the command succeeds, the virtual function is removed from the ldg1 domain. When ldg1 is restarted, the specified virtual function no longer appears in that domain.

If the domain that has the virtual function is the primary domain, you must initiate a delayed reconfiguration.

SR-IOV: Rebooting the Root Domain

Take care when rebooting the primary domain. See Rebooting the Root Domain. As with PCIe slots in the I/O domain, the concerns that are described in this section also pertain to the virtual functions that are assigned to an I/O domain.

Using an SR-IOV Virtual Function to Create an I/O Domain

The following procedure explains how to create an I/O domain that includes PCIe SR-IOV virtual functions.

How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It

Plan ahead to minimize the number of reboots of the primary domain, which minimizes the downtime.

Identify an SR-IOV physical function to share with an I/O domain that uses the SR-IOV feature.
```
primary# ldm list-io
```
Obtain detailed information about the physical function that you want to share.
```
primary# ldm list-io -l pf-name
```
Initiate a delayed reconfiguration on the root domain.
```
primary# ldm start-reconf primary
```
Enable I/O virtualization for a PCIe bus.
```
primary# ldm set-io iov=on bus
```
Create one or more virtual functions for the physical function.
```
primary# ldm create-vf pf-name
```
You can run this command for each virtual function that you want to create. If you perform these commands as a batch, you only need to reboot the primary domain one time.
Stop any I/O domain that has a PCIe endpoint device or a virtual function assigned to it.
```
primary# ldm stop ldom
```
Reboot the root domain.
```
primary# reboot
```
View the list of available virtual functions on the primary domain.
```
primary# ldm list-io
```
Assign the virtual function to a domain.
```
primary# ldm add-io vf-name ldom
```

Bind and start the domain.

primary# ldm bind ldom
primary# ldm start ldom

Verify that the virtual function is available on the I/O domain.
The following Oracle Solaris 11 command shows the availability of the virtual function:
```
guest# dladm show-phys
```

Example 6-6 Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It

The following example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.

The following ldm list-io output shows that the /SYS/MB/NET0/IOVNET.PF0 physical function is available:

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary  
pci_1                                     BUS    pci_1    primary  
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

The following command shows additional details about the /SYS/MB/NET0/IOVNET.PF0 physical function, which includes the maximum number of virtual functions that can be created:

primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0]
    maxvfs = 7

The following command creates a virtual function called /SYS/MB/NET0/IOVNET.PF0.VF0 for the /SYS/MB/NET0/IOVNET.PF0 physical function:

primary# ldm start-reconf primary
primary# ldm set-io iov=on pci_0
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0

------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Because the ldg1 I/O domain has a PCIe endpoint device that was created by using the DIO feature, you must stop the ldg1 domain and reboot the primary domain, as follows:

primary# ldm stop ldg1
primary# reboot

The following command verifies that the new /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function exists:

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary  IOV      
pci_1                                     BUS    pci_1    primary  
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary           
/SYS/MB/NET0/IOVNET.PF0.VF0               VF     pci_0

The following command assigns the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain:

primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

The following commands bind and restart the ldg1 domain:

primary# ldm bind ldg1
primary# ldm start ldg1

The following command verifies that the virtual function is available for use:

guest# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         0      unknown   vnet0
net1              Ethernet             up         1000   full      igbvf0

Advanced SR-IOV Topics

This section describes several advanced topics that arise when you use PCIe SR-IOV-capable I/O devices.

Booting an I/O domain by using an SR-IOV virtual function. An SR-IOV virtual function provides similar capabilities as any other type of PCIe device, such as the ability to use a virtual function as a logical domain boot device. For example, a network virtual function can be used to boot over the network to install the Oracle Solaris OS in an I/O domain.

Note - When booting the Oracle Solaris OS from a virtual function device, verify that the Oracle Solaris OS being loaded has virtual function device support. If so, you can continue with the rest of the installation, as planned.
Maximum number of I/O domains and virtual functions supported. The PCIe endpoint devices and SR-IOV virtual functions from a particular PCIe bus can be assigned up to a maximum of 15 domains. The PCIe resources, such as interrupt vectors for each PCIe bus, are divided among the root domain and I/O domains. As a result, the number of devices that you can assign to a particular I/O domain is also limited. So, ensure that you do not assign a large number virtual functions to the same I/O domain. For a description of the problems related to SR-IOV, see Oracle VM Server for SPARC 3.0 Release Notes.

SR-IOV Device-Specific Properties

SR-IOV physical function device drivers can export device-specific properties. These properties can be used to tune the resource allocation of both the physical function and its virtual functions. For information about the properties, see the man page for the physical function driver, such as the igb(7D) and ixgbe(7D) man pages.

The ldm list-io -d command shows device-specific properties that are exported by the specified physical function device driver. Each property has a name, brief description, default value, maximum values, and one or more of the following flags:

P: Applies to a physical function
V: Applies to a virtual function
R: Read-only or informative parameter only

primary# ldm list-io -d pf-name

Use the ldm create-vf or ldm set-io command to set the read-write properties for a physical function or a virtual function. Note that to set a device-specific property, you must initiate a delayed reconfiguration.

The following example shows the device-specific properties that are exported by the on-board Intel 1-Gbps SR-IOV device:

primary# ldm list-io -d /SYS/MB/NET0/IOVNET.PF0
Device-specific Parameters
--------------------------
max-config-vfs
    Flags = PR
    Default = 7
    Descr = Max number of configurable VFs
max-vf-mtu
    Flags = VR
    Default = 9216
    Descr = Max MTU supported for a VF
max-vlans
    Flags = VR
    Default = 32
    Descr = Max number of VLAN filters supported
pvid-exclusive
    Flags = VR
    Default = 1
    Descr = Exclusive configuration of pvid required
unicast-slots
    Flags = PV
    Default = 0 Min = 0 Max = 24
    Descr = Number of unicast mac-address slots

The following example sets the unicast-slots property to 8:

primary# ldm create-vf unicast-slots=8 /SYS/MB/NET0/IOVNET.PF0

Advanced Network Configuration for Virtual Functions

SR-IOV virtual functions can only use the MAC addresses that are assigned by the Logical Domains Manager. If you use other Oracle Solaris OS networking commands to change the MAC address on the I/O domain, the commands might fail or might not function properly.
At this time, link aggregation of SR-IOV network virtual functions in the I/O domain is not supported. If you attempt to create a link aggregation, it might not function as expected.
You can create virtual I/O services and assign them to I/O domains. These virtual I/O services can be created on the same physical function from which virtual functions are also created. For example, you can use an on-board 1-Gbps network device (net0 or igb0) as a network back-end device for a virtual switch and also create virtual functions from the same physical function device.

Creating VNICs on SR-IOV Virtual Functions

The creation of Oracle Solaris 11 VNICs is supported on SR-IOV virtual functions. However, the number of VNICs that is supported is limited to the number of alternate MAC addresses (alt-mac-addrs property) assigned to the virtual function. So, ensure that you assign a sufficient number of alternate MAC addresses when you use VNICs on the virtual function. Use the ldm create-vf or ldm set-io command to set the alt-mac-addrs property with the alternate MAC addresses.

The following example shows the creation of four VNICs on an SR-IOV virtual function. The first command assigns alternate MAC addresses to the virtual function device. This command uses the automatic allocation method to allocate four alternate MAC addresses to the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function device:

primary# ldm set-io alt-mac-addrs=auto,auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0

The next command starts and boots the Oracle Solaris 11 OS in the I/O domain. In this example, ldg1 is the I/O domain:

primary# ldm start ldg1

The following command uses the Oracle Solaris 11 dladm command in the In the guest domain to create four VNICs. Note that attempts to create more VNICs than are specified by using alternate MAC addresses will fail.

guest# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         0      unknown   vnet0
net1              Ethernet             up         1000   full      igbvf0
guest# dladm create-vnic -l net1 vnic0
guest# dladm create-vnic -l net1 vnic1
guest# dladm create-vnic -l net1 vnic2
guest# dladm create-vnic -l net1 vnic3
guest# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net0                phys      1500   up       --
net1                phys      1500   up       --
vnic0               vnic      1500   up       net1
vnic1               vnic      1500   up       net1
vnic2               vnic      1500   up       net1
vnic3               vnic      1500   up       net1

Skip Navigation Links
Exit Print View
	Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC